Ingesting Massive DB with 600 Million Vertices and 14 Billion Edges

[keygenx]

Ingesting Massive DB with 600 Million Vertices and 14 Billion Edges

Context & Background

I am in the process of ingesting a massive graph dataset containing over 600 Million vertices (663M actual count) and billions of relationships/edges (scaling towards ~14 Billion total connections) into a graph database.

I am posting this to share my hardware setup, data formats, 4-phase ingestion architecture, performance benchmarks, and real-world technical hurdles—and to ask for community feedback on how to optimize and strategize high-throughput graph ingestion at this scale.

---

Hardware & Configuration Setup

• CPU: AMD EPYC 7532 (32 Cores, 64 Threads, 2.40 GHz base clock)
• RAM: 256 GB DDR4 3200 MHz
• Database Storage: Dedicated Samsung 990 Evo Plus 4TB NVMe SSD (formatted with XFS filesystem)
• Intermediate Key-Value Map Storage: Secondary NVMe SSD (physically separate drive, formatted with EXT4 filesystem)
• Source Snapshot Storage: Compressed .gz files on a ZFS storage pool backed by spinning HDDs (separated from NVMe database storage to avoid disk I/O contention)
• JVM Heap & GC Config: Fixed 64 GB Heap (-Xms64G -Xmx64G) to prevent dynamic JVM allocation overhead, configured with G1 Garbage Collector (-XX:+UseG1GC) and parallel reference processing (-XX:+ParallelRefProcEnabled).
• Storage Partitioning Config: Default storage bucket count set to 16 buckets per vertex type (typeDefaultBuckets=16) to allow concurrent parallel file reads and writes across storage partitions.
• Software Stack: Python 3.14 (parallel processing via joblib, rapid JSON serialization via orjson), high-performance local key-value engine (RocksDB via rocksdict) for entity ID mapping, gRPC / HTTP connection drivers.

---

Source Data Format

The dataset snapshot consists of raw compressed .gz files containing newline-delimited JSON (JSONL).

• Vertices (~663 Million Total): Distributed across multiple entity types (with the primary entity type comprising ~500M+ records).
• Edges (Billions of Links): Relational linkages embedded within JSON entities (e.g. citations, entity relationships, affiliations, category associations).

---

4-Phase Ingestion Architecture

To bypass expensive database index lookups during edge insertion, I designed a 4-Phase Ingestion Pipeline utilizing an external key-value store to map natural string IDs to internal database record pointers.

+------------------------------------+
|  Source Data (.gz JSONL on HDD)    |
+------------------------------------+
                  |
                  v
[ Phase 1: Vertex Bulk Ingestion ] ----> [ Graph Database (663M Vertices) ]
                  |                                    |
                  v                               .----' (16 Storage Buckets)
[ Phase 2: Build External KV Map ] <-------------
                  |
                  v
       ( RocksDB Local Store )
       ( string ID -> internal record pointer )
                  |
                  v
[ Phase 3: Deduplication ]
                  |
                  v
[ Phase 4: Edge Direct Bulk Load ] ----> [ Graph Database (Edges via Record Pointers) ]

Phase 1: Vertex Ingestion

• Strategy: 12 parallel worker processes stream and parse compressed .gz files. Vertices are ingested in batches (16,384 memory buffer size, 1,024–4,096 SQL/API statements per execution chunk) into the database across the 16 partition storage buckets.
• Performance: Loaded 663 Million vertices in ~22 hours (average throughput ~8,370 vertices/sec).
• Disk Profile: Target NVMe writes averaged 18.6 MB/s (peaking at 60.0 MB/s), while source HDD reads remained steady around 9.1 MB/s.

Phase 2: Building the External ID-to-Record Pointer Map

• Creating edges by querying indexed fields (WHERE id = '...') at billion-edge scale introduces massive query overhead.
• Strategy: Parallel workers scroll the database storage partitions across all 16 default storage buckets per type in parallel using streaming gRPC channels to extract (external_string_id, internal_record_pointer).
• Performance: Database streaming read throughput reached ~330,000 records/sec. End-to-end throughput (including record parsing and inserting into the local RocksDB KV map) sustained 120,000–138,000 records/sec, mapping all 663M vertices in 1 hour 20 minutes.
• Storage Separation: Reading database files from NVMe Drive 1 (280 MB/s average read) while writing the KV store to NVMe Drive 2 (19.4 MB/s average write) completely eliminated read/write disk contention.

Phase 3: Deduplication & Unique Index Creation

• Optimistic parallel vertex inserts without pre-existing unique indexes can occasionally create duplicate nodes under concurrent thread contention.
• Strategy: The KV map flags entity IDs pointing to multiple record pointers. 4 asynchronous workers processed deletions, removing 23 million duplicate vertices in ~50 minutes (~7,670 deletions/sec).

Phase 4: Direct Edge Ingestion

• Strategy: Parallel CPU workers read edge files, parse source/target IDs, resolve them to direct internal record pointers via the local RocksDB store, and enqueue batches.
• Single-Ingester Pattern & GraphBatchLoad API: Rather than running multiple concurrent client streams (which caused severe server-side partition lock contention), a single gRPC streaming client utilizes the dedicated GraphBatchLoad streaming API (graph_batch_load_stream) to consume prepared batches from a shared queue and stream edges directly into the database using internal record pointers, bypassing index lookups completely.

---

Issues & Challenges Encountered

1. Storage Partition & Page Lock Contention:

   • Symptom: Running multiple parallel writers streaming edge/vertex batches concurrently caused frequent lock contention and page version mismatches (ConcurrentModificationException).
   • Fix: Shifted to a decoupled architecture—multi-process CPU workers for data parsing/ID resolution feeding a single streaming writer client with exponential backoff retries.

2. JVM Heap Accumulation & GC Pause Spikes (GraphBatchLoad Stream):

   • Symptom: Holding a single gRPC GraphBatchLoad streaming RPC open continuously over long multi-hour runs caused uncommitted incoming edge structures to accumulate in JVM heap memory. This triggered severe Garbage Collection (GC) latency spikes and degraded ingestion throughput over time. Under heavy concurrency, stream threads could also trigger server-side async executor shutdown warnings (Async executor has been shut down).
   • Fix: Locked JVM heap allocations at 64 GB (-Xms64G -Xmx64G) with G1GC (-XX:+UseG1GC) and implemented periodic stream recycling every 4,000,000 records. Completing and re-establishing the GraphBatchLoad RPC forces the engine to flush memory buffers, reclaim JVM heap memory, and reset HTTP/2 transport flow-control windows.

3. Request Buffer Overflow & Packet Truncation:

   • Symptom: Excessively large batch sizes (>32,000 records) produced 40MB–50MB payloads that exceeded server HTTP/gRPC request buffer limits, triggering payload truncation and JSON parsing errors.
   • Fix: Reduced and capped batch sizes to 1,024–4,096 records per request.

4. Abrupt Shutdowns & Lock Acquisition Failures:

   • Symptom: Process or container SIGKILLs during active batch execution left lock state dirty, leading to page lock acquisition timeouts (InterruptedIOException) on startup.
   • Fix: Enforced graceful container teardown (stop -t 60) and enabled automated Write-Ahead Log (WAL) recovery on restart.

---

Please Advise

During my scale testing, I found myself caught in a clear technical trade-off:

• Large Batch Sizes $\rightarrow$ Memory & Buffer Pressure: Pushing very large batch sizes to the GraphBatchLoad API causes server heap accumulation, memory pressure, stream degradation over multi-hour runs, and request buffer overflows (>32k batch sizes causing packet truncation).
• Multiple Parallel Small-Batch Streams $\rightarrow$ Lock Contention: Splitting ingestion into smaller batches across multiple parallel GraphBatchLoad streaming workers triggers severe storage bucket/page lock collisions (ConcurrentModificationException), thread pool exhaustion, and async executor crashes on the database server.

I am seeking advice and perspectives from practitioners on the optimal database configuration and ingestion strategy to resolve this trade-off when loading multi-billion-edge graphs:

1. Batch Sizing & Stream Cadence Strategy:

   • What batch size ranges and stream recycling cadences provide the best balance between JVM heap stability and write throughput when streaming edges via batch APIs?

2. Concurrency & Lock Contention Avoidance:

   • How can I configure or structure parallel GraphBatchLoad ingestion streams without triggering storage partition lock collisions? Is offline sorting/grouping of edges by target vertex storage bucket prior to ingestion an effective pattern for lock-free parallel loading?

3. Database Engine & JVM Memory Configuration:

   • What database memory allocation settings (heap size vs off-heap page cache vs WAL transaction flush settings vs default bucket partition counts per document type) have yielded the highest sustained write throughput when the total graph dataset size far exceeds available system RAM (256 GB)?

4. Alternative Bulk Ingestion Strategies:

   • Beyond direct record pointer resolution, are there specific engine flags, bulk-loader modes, or offline database file generation techniques recommended for ingesting 10B+ edge datasets cleanly?

Any insights, configuration recommendations, or strategy suggestions would be greatly appreciated!

[lvca]

Thanks for one of the most detailed reports we have had. I went through the code paths your pipeline exercises, and the headline is that the trade-off you describe is not real: both horns are the same root cause, and neither is on the batch-size axis. Four issues came out of it.

What is actually happening

Heap growth is proportional to stream length, not to batch size. GraphBatch keeps two caches that map every distinct vertex an edge touches, and neither is cleared until close(). At roughly 80-90 bytes per entry in each of the two maps, a stream that touches 100M distinct vertices holds 16-18 GB before anything else. The deferred incoming-edge buffer adds 36 bytes per edge and is drained only at close(). Meanwhile batchSize sizes the outgoing buffer, which is reset every flush, so the one knob you reached for is the one knob that cannot affect this. Your "recycle every 4M records" is not a workaround, it is currently the only way the component works over a long stream. Tracked in #5664.

Your Async executor has been shut down has a specific cause. Every flush calls two setters on the shared async executor, and each of them tears down and respawns the entire worker pool. That is four full pool recreations per flush, 63 threads each on your EPYC, roughly 140k flushes into your load. Any other user of the async executor, including a second GraphBatchLoad stream, has its in-flight and queued tasks force-exited by the first batch's flush. Tracked in #5665.

Your single-writer fallback was right for a reason you could not have known. Two concurrent GraphBatch instances on one database can silently orphan edges: each defers the vertex head-chunk pointer to its own close(), so both see a null head for the same vertex, both create a segment chain, and the last one to close wins. The loser's edges are persisted and reachable from nothing, with no exception anywhere. Tracked in #5666, along with the fact that nothing currently stops you from creating them.

What to change, in order of impact

1. Declare the property-free edge types LIGHTWEIGHT.

CREATE EDGE TYPE Cites LIGHTWEIGHT

Right now you are creating 14 billion edge records you do not need, which is hundreds of GB of disk and a large share of your write time.

Use a build that includes the recent lightweight-edge identity work (on main, so 26.8.1-SNAPSHOT or later). Before it, a lightweight edge was addressed by a per-type marker RID rather than a real identity, which broke Cypher multi-hop patterns over lightweight chains and made the JSONL exporter drop them silently. For a traversal workload on a 14B-edge database, I would not have suggested this on an older snapshot.

Two conditions, per type. The edges carry no properties, now enforced with a clear error rather than silently promoted to a record. And parallel edges between the same pair are not meaningful, since identity is the triple (type, out, in). Citations and category associations are naturally at most one per pair; anything that needs a date range or a weight stays a regular type. The declaration is per type, so you can mix.

Two notes for your pipeline specifically. Since you retry and recycle streams, budget for a CHECK DATABASE pass afterwards: duplicates buffered inside one flush are reported for free, but a duplicate split across flushes or streams only surfaces there, counted under duplicateLightEdges. It reads fine and each delete removes one copy, it is a modelling advisory rather than damage. And do not declare UNIQUE for this load: on a lightweight type it scans the source vertex's edge list, O(degree), on every insert. You already have RocksDB in the pipeline, so dedupe there.

2. Move the vertex phase onto the same GraphBatchLoad stream. 8,370 vertices/sec is well below the bulk vertex path, and vertices sent with temp_id come back as id_mapping in the result, so phases 2 and 3 (2h10m plus 23M duplicates) largely stop existing. Those duplicates only appeared because the vertex phase ran concurrently without a unique index. A single writer cannot produce them.

3. Give the page cache real memory. arcadedb.maxPageRAM defaults to a quarter of the heap, so with -Xmx64G you are running a 16 GB cache against a multi-TB working set. Pages are heap ByteBuffers, so the heap is the ceiling for the cache. On a 256 GB box, consider a much larger heap with maxPageRAM set explicitly.

4. Edge-connect parallelism is capped by bucket count, not by cores. Work is dispatched as bucketId % parallelLevel, so your 16 buckets per type use 16 of 63 async slots. For the current run, setting arcadedb.asyncWorkerThreads=16 at least stops you paying for 47 idle threads on every pool respawn. For the next load, buckets per type near the core count is the setting you want; changing it on the existing database means a repartition.

5. Keep recycling the stream until #5664 lands. 4M records is a reasonable cadence.

One knob to not reach for: arcadedb.graph.edgeListInitialChunkSize (and the equivalent GraphBatch option) looks like it should matter at your average degree of ~21, but with parallelFlush on, which is the default, new segments are already created exactly sized for the edges landing in that flush. It only bites the pre-allocating single-vertex API.

One thing to plan for that you have not hit yet

A graph loaded entirely through the bulk path never gets the striped super-node layout, so your category and affiliation hubs end up as chains of thousands of 8 KB segments and traverse badly, after a load that looked completely successful. That is #5667. Worth knowing before you commit to the full 14B run rather than after.

On the payload truncation

The gRPC server allows 100 MB by default (grpc.maxMessageSize), and gRPC does not produce JSON parse errors, so that symptom looks like it came from the HTTP path rather than from GraphBatchLoad. Note the client-side default inbound limit is 4 MB, which is the more likely constraint on the response side if you use temp_id at scale. Worth confirming which client hit it.

Finally

At 14B edges the wire protocol itself is a real tax. Running the loader embedded in a JVM against the engine directly removes the protobuf conversion, the per-stream maps and the message-size limits entirely. There is also a declarative GraphImporter in the integration module with a JSONL source that does the two-pass vertices-then-edges shape you built by hand.

Please keep posting numbers as you go, this is exactly the scale where the interesting problems live.

[keygenx]

Thank you for the thorough review and analysis of the issues.

To help document these issues further and potentially assist with debugging, here are the actual log snippets and stack traces recorded across the different failure modes encountered during ingestion:

---

1. Async executor has been shut down (Tracked in #5665)

This error surfaced across both transport layers under heavy batch loads:

HTTP Batch Endpoint (PostBatchHandler):

SEVER [PostBatchHandler] <ArcadeDB_0> Error on command execution (PostBatchHandler): Async executor has been shut down; cannot schedule asynchronous task Transaction(com.arcadedb.graph.GraphBatch$$Lambda/0x00007f95f90c41f8@47eba8c)

gRPC Streaming Endpoint (ArcadeDbService/GraphBatchLoad):

2026-07-19 21:19:02.603 WARNI [GrpcLoggingInterceptor] gRPC call failed: com.arcadedb.grpc.ArcadeDbService/GraphBatchLoad - Status{code=INTERNAL, description=graphBatchLoad: Async executor has been shut down; cannot schedule asynchronous task Transaction(com.arcadedb.graph.GraphBatch$$Lambda/0x00007f60b3512eb0@32716a11), cause=null} (157ms)

---

2. HTTP Payload Buffer Overflow & Truncation

Confirming your diagnosis regarding the HTTP vs gRPC transport limits: when testing larger batch sizes (e.g. --batch-size 32768) over the HTTP batch API endpoint, NDJSON payloads grew to ~40–50 MB. This exceeded HTTP server packet buffers, causing stream truncation and JSON parsing failures on the server:

Malformed JSON at line ...: Invalid JSON object format

---

3. Page Lock Interruption (InterruptedIOException) & Inaccessible Database During Querying

When a worker process or container was abruptly interrupted during active GraphBatch execution, thread lock state became dirty, leading to lock acquisition failures on server startup when gathering bucket statistics:

 2026-07-18 09:18:02.630 WARNI [LocalBucket] Error on gathering statistics on bucket 'authors_0_out_edges'
 java.io.InterruptedIOException: Interrupted while acquiring I/O lock for page PageId(test/50/107811)
        at com.arcadedb.engine.PageManager.concurrentPageAccess(PageManager.java:635)
        at com.arcadedb.engine.PageManager.loadPage(PageManager.java:608)
        at com.arcadedb.graph.GraphBatch.getOrCreateOutEdgeChunk(GraphBatch.java:1589)

Noticeably, even after container restart logs reported that WAL recovery completed successfully:

2026-07-18 09:28:07.485 WARNI [LocalDatabase] <ArcadeDB_0> Database 'test' was not closed properly last time
2026-07-18 09:28:07.488 WARNI [TransactionManager] <ArcadeDB_0> Started recovery of database 'test'
2026-07-18 09:28:07.505 WARNI [TransactionManager] <ArcadeDB_0> Recovery of database 'test' completed

The database still remained inaccessible to queries for several hours.

---

4. ConcurrentModificationException During Batched Deletions (DELETE FROM [rids])

During Phase 3 duplicate removal using 4 parallel gRPC workers running batched deletion queries (DELETE FROM [rids]), two distinct concurrency error patterns emerged:

A. Page Version Mismatch During 1st-Phase Transaction Commit:

2026-07-18 17:00:49.074 SEVER [ArcadeDbGrpcService] ERROR in executeCommandInternal
com.arcadedb.exception.ConcurrentModificationException: Concurrent modification on page PageId(test/46/1159724) in file 'works_15.46.65536.v0.bucket' (current v.6 <> database v.7). Please retry the operation (threadId=1391)
        at com.arcadedb.engine.PageManager.checkPageVersion(PageManager.java:358)
        at com.arcadedb.database.TransactionContext.commit1stPhase(TransactionContext.java:1064)
        at com.arcadedb.database.TransactionContext.commit(TransactionContext.java:195)
        at com.arcadedb.database.LocalDatabase.lambda$commit$5(LocalDatabase.java:522)
        at com.arcadedb.database.LocalDatabase.executeInReadLock(LocalDatabase.java:1778)
        at com.arcadedb.database.LocalDatabase.commit(LocalDatabase.java:516)
        at com.arcadedb.server.ServerDatabase.commit(ServerDatabase.java:175)
        at com.arcadedb.server.grpc.ArcadeDbGrpcService.executeCommandInternal(ArcadeDbGrpcService.java:694)
        at com.arcadedb.server.grpc.ArcadeDbServiceGrpc$MethodHandlers.invoke(ArcadeDbServiceGrpc.java:1184)

B. Multi-Page Record Modification During Read / Edge Cleanup:

2026-07-18 17:00:49.501 WARNI [GraphEngine] Error on deleting outgoing edges connected to vertex #31:2442876940
com.arcadedb.exception.ConcurrentModificationException: Multi-page record #31:2442876940 was modified during read after 3 retries. Please retry the operation
        at com.arcadedb.engine.LocalBucket.loadMultiPageRecord(LocalBucket.java:1567)
        at com.arcadedb.engine.LocalBucket.getRecordInternal(LocalBucket.java:762)
        at com.arcadedb.engine.LocalBucket.getRecord(LocalBucket.java:251)
        at com.arcadedb.database.ImmutableDocument.checkForLazyLoading(ImmutableDocument.java:216)
        at com.arcadedb.graph.ImmutableVertex.checkForLazyLoading(ImmutableVertex.java:251)
        at com.arcadedb.graph.ImmutableVertex.getOutEdgesHeadChunk(ImmutableVertex.java:118)
        at com.arcadedb.graph.GraphEngine.deleteVertex(GraphEngine.java:451)
        at com.arcadedb.database.LocalDatabase.deleteRecordNoLock(LocalDatabase.java:1225)
        at com.arcadedb.database.LocalDatabase.lambda$deleteRecord$23(LocalDatabase.java:1175)
        at com.arcadedb.database.LocalDatabase.executeInReadLock(LocalDatabase.java:1778)
        at com.arcadedb.database.LocalDatabase.deleteRecord(LocalDatabase.java:1174)
        at com.arcadedb.query.sql.executor.DeleteStep$1.next(DeleteStep.java:49)

---

5. Transient Record Lookup Failure During gRPC GraphBatchLoad

During Phase 4 edge ingestion, occasional batch loads failed with record lookup errors:

2026-07-20 02:30:00.346 WARNI [GrpcLoggingInterceptor] gRPC call failed: com.arcadedb.grpc.ArcadeDbService/GraphBatchLoad - Status{code=INTERNAL, description=graphBatchLoad: Error on lookup of record #1:3254642714, cause=null} (20255ms)

Note on this error: Manually running SELECT FROM [#1:3254642714] directly against ArcadeDB confirmed that the record did exist and was fully resolvable (ruling out a stale/pruned RID in the RocksDB map). The ~20-second latency before the error surfaces suggests a transient lookup/page-lock timeout inside GraphBatchLoad under concurrent write load.

---

Thanks again for the fantastic response and for logging those fixes! Resources are currently stretched very thin as I am bootstrapping a company, but I plan to restart experimentation in a week or two with the suggested fixes applied. I'll keep documenting my findings here and we can continue the discussion.