Skip to content

Fix the V8+WASIX large-payload memory leak and the host-heap corruption abort - #53

Merged
syrusakbary merged 2 commits into
mainfrom
fix/v8-large-payload-leak
Aug 10, 2026
Merged

Fix the V8+WASIX large-payload memory leak and the host-heap corruption abort#53
syrusakbary merged 2 commits into
mainfrom
fix/v8-large-payload-leak

Conversation

@Arshia001

Copy link
Copy Markdown
Member

Two independent bugs on the V8/N-API WASIX lane, both surfaced by moving large HTTP request bodies through wasmer/edgejs-v8. Neither affects quickjs.

1. Guest-heap chunk allocators leaked ~2 MiB of host metadata per 1 MiB chunk

GuestHeap::claim_chunk_locked built each chunk's allocator with offset_allocator::Allocator::new(len / UNIT). That constructor hard-codes max_allocs = 128 * 1024 regardless of chunk size, and its reset() eagerly allocates nodes: vec![Node; max_allocs] (28 bytes each) plus a u32 free-list entry. So every 1 MiB chunk of guest linear memory cost roughly 2 MiB of host metadata. Chunks are claimed as a workload's peak grows and are never released, so that overhead accumulated for the life of the process — host RSS grew about three times the guest memory being managed.

heaptrack named it outright on a run that moved only 240 MiB:

272.63M leaked over 130 calls from
offset_allocator::Allocator::with_max_allocs

130 calls = 130 chunks.

Fix: budget one node per 256 bytes of chunk (clamped to [1024, 32768]), which leaves ~2x headroom for the ~10 KiB buffers this heap actually serves. Exhausting a chunk's nodes is graceful — allocate returns None and try_chunks falls through to another chunk — so an under-estimate costs a little fragmentation, not correctness.

Measured on the payload load test (echo-buffer, 10 MiB bodies, 1.95 GiB moved):

before after
Net RSS growth 368 MB 134 MB
Host-anon growth ~20 MB/GiB ~1 MB/GiB
Per-round retention, last 3 rounds 0.2 / 0.0 / 5.2 MB 0.4 / -0.1 / 0.1 MB

2. The external-backing-store hint set was mutated from V8 sweeper threads

V8 destroys backing stores from its ArrayBufferSweeper, which runs on background threads, so ExternalBackingStoreDeleter executes concurrently with the main thread creating new hints. It erased from napi_env__::external_backing_store_hints — a std::unordered_set — and deleted the hint, with no synchronization against the two insert sites or against teardown. Concurrent mutation of the set corrupts the host heap: glibc aborts with malloc_consolidate(): unaligned fastbin chunk detected a few hundred requests into a payload-heavy workload, killing the runtime mid-request.

GuestHeap already takes its own mutex precisely because "V8 GC threads free backing stores off the JS thread". This container was missed.

Fix: guard the set with a process-global mutex — global rather than a member because a deleter must read hint->env before it knows which env to lock, and that read itself races teardown. The deleter now reads env, unlinks, and claims finalize_cb in one critical section, then runs the finalizer outside the lock (finalizers re-enter N-API). Teardown detaches the whole set under the lock and collects the claimed callbacks, so exactly one path ever runs a given finalizer and teardown never touches a hint a concurrent deleter may already be freeing.

Evidence. The abort is too flaky to A/B directly, so the race was measured rather than waited for:

  • Instrumenting the deleter with the calling thread id showed it firing on a non-main thread within the first round of the load test.
  • A try_lock counter on the new mutex recorded 300+ genuine concurrent accesses in a single five-round run — each one a moment the old lock-free code was mutating the set from two threads at once.

With a config that does reproduce the abort (echo-buffer, 256 KiB bodies, concurrency 8, five rounds): 2 of 6 runs died before this change (one with the malloc_consolidate signature), 0 of 28 after.

Testing

  • 38/38 wasmer-napi lib tests green, including two new guest_heap tests covering the node budget and a chunk's real capacity in ~10 KiB buffers.
  • 150-round / ~15 GiB soak: GuestHeap claims 72 chunks total, 66 of them in the first 25 rounds, then 2/2/0/1 per subsequent 25-round block — the linear-memory footprint converges. A separate 150-round / 29.3 GiB run has total RSS growth fitting a logarithmic curve 2.4x better than linear.
  • All instrumentation used during the investigation was removed; this branch is two focused commits.

Every GuestHeap chunk built its offset allocator with `Allocator::new`, which
sizes the node arena for 128Ki allocations no matter how big the chunk is. Its
`reset` allocates that eagerly — a 28-byte Node plus a u32 free-list slot per
allocation — so each 1 MiB chunk of guest linear memory cost roughly 2 MiB of
HOST metadata. Chunks are claimed as a workload's peak grows and are never
released, so that overhead accumulated for the life of the process.

This was the dominant term in the V8+WASIX large-payload leak. heaptrack
attributes 272 MB of the 343 MB retained by a 240 MiB echo run to
`offset_allocator::Allocator::with_max_allocs` over 130 calls — one per chunk.
Host RSS grew about three times the guest memory being managed, which is why
the leak appeared to live half in linear memory and half on the host heap.

Budget one node per 256 bytes of chunk instead (clamped), leaving ~2x headroom
for the ~10 KiB buffers this heap actually serves. Exhausting a chunk's nodes
is graceful — `allocate` returns None and `try_chunks` falls through to another
chunk — so an under-estimate costs a little fragmentation, not correctness.

Measured on the payload load test (echo-buffer, 10 MiB bodies, 1.95 GiB moved,
wasmer/edgejs-v8 under wasmer#6850): net RSS growth 368 MB -> 134 MB, host
anonymous RSS growth 236 MB -> 16 MB, and per-round retention now settles to
0.4 / -0.1 / 0.1 MB over the last three rounds instead of still stepping.
… threads

V8 destroys backing stores from its ArrayBufferSweeper, which runs on
background threads, so ExternalBackingStoreDeleter executes concurrently with
the main thread creating new hints. The deleter erased from
napi_env__::external_backing_store_hints -- a std::unordered_set -- and deleted
the hint, with no synchronization against the two insert sites or against
teardown. Concurrent mutation of the set corrupts the host heap: glibc aborts
with "malloc_consolidate(): unaligned fastbin chunk detected" a few hundred
requests into a payload-heavy workload, killing the runtime mid-request.

GuestHeap already takes its own mutex precisely because "V8 GC threads free
backing stores off the JS thread"; this container was missed.

Guard the set with a process-global mutex -- global rather than a member
because a deleter must read hint->env before it knows which env to lock, and
that read itself races teardown. The deleter now reads env, unlinks, and claims
finalize_cb in one critical section, then runs the finalizer outside the lock
since finalizers re-enter N-API. Teardown detaches the whole set under the lock
and collects the claimed callbacks, so exactly one of the two paths ever runs a
given finalizer and teardown never touches a hint a concurrent deleter may
already be freeing.

Evidence: instrumenting the deleter with the calling thread id showed it firing
on a non-main thread within the first round of the payload load test, and a
try_lock counter on the new mutex recorded 300+ genuine concurrent accesses in
a single five-round run -- each one a moment the old lock-free code was
mutating the set from two threads at once. On the payload load test
(echo-buffer, 256 KiB bodies, concurrency 8, five rounds), the runtime died in
2 of 6 runs before this change -- one with the malloc_consolidate signature --
and in 0 of 18 runs after.
@syrusakbary
syrusakbary merged commit f3c56b3 into main Aug 10, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants