bmalloc: add 1ms backoff and retry cap to SYSCALL/PAS_SYSCALL EAGAIN loops

[coleleavitt]

Summary

The SYSCALL and PAS_SYSCALL macros in bmalloc retry syscalls on EAGAIN in a zero-delay tight loop. When madvise(MADV_DONTDUMP) returns EAGAIN due to kernel mmap_write_lock contention, this causes 100% CPU usage across all GC threads — effectively freezing the application.

This PR adds usleep(1000) backoff (1ms) and caps retries at 100 (100ms total).

Root Cause Analysis

LLM streaming / heavy allocation workload
  → JSC allocation-triggered GC fires
  → GC sweep calls bmalloc vmDeallocatePhysicalPages() thousands of times
  → Each call does TWO madvise syscalls: MADV_DONTNEED + MADV_DONTDUMP
  → MADV_DONTDUMP requires kernel mmap_write_lock (unlike MADV_DONTNEED which only needs read lock)
  → Multiple GC threads contend on single process-wide mmap_write_lock
  → Kernel returns EAGAIN (VMA split/merge allocation failure under memory pressure)
  → SYSCALL macro retries in ZERO-DELAY infinite loop: while((x)==-1 && errno==EAGAIN){}
  → 250K+ madvise calls/sec/thread, 100% CPU, application frozen

The Smoking Gun

BSyscall.h (before):

#define SYSCALL(x) do { \
    while ((x) == -1 && errno == EAGAIN) { } \
} while (0);

pas_utils.h (before):

#define PAS_SYSCALL(x) do { \
    while ((x) == -1 && errno == EAGAIN) { } \
} while (0)

Zero-delay infinite retry. No backoff, no sleep, no yield, no retry cap.

The Fix

// BSyscall.h
#define SYSCALL(x) do { \
    int _syscall_tries = 0; \
    while ((x) == -1 && errno == EAGAIN) { \
        if (++_syscall_tries > 100) break; \
        usleep(1000); \
    } \
} while (0);

// pas_utils.h
#define PAS_SYSCALL(x) do { \
    int _pas_syscall_tries = 0; \
    while ((x) == -1 && errno == EAGAIN) { \
        if (++_pas_syscall_tries > 100) break; \
        usleep(1000); \
    } \
} while (0)

Why This Approach

• 1ms fixed delay is ~1000× longer than kernel lock hold time — more than enough for contention to clear
• 100 retry cap (100ms total) prevents infinite loops under pathological conditions; madvise failures here are advisory, not fatal
• Zero fast-path impact — the while body is dead code when the syscall succeeds
• Matches existing Windows precedent — virtual_alloc_with_retry() in libpas/pas_page_malloc.c already uses Sleep(50ms) with 10 max retries
• Consistent with tcmalloc — Google's tcmalloc uses bounded retries (3 attempts) for expensive madvise operations (source)

Why NOT sched_yield()

Per Red Hat RHEL-RT Tuning Guide: sched_yield can reschedule immediately (busy loop) or after long delay — unpredictable behavior. usleep(1000) provides deterministic 1ms backoff.

Blast Radius

17 callsites affected (all madvise/mprotect/mincore — all benefit from this fix):

• 6 in bmalloc/VMAllocate.h (madvise calls in vmDeallocatePhysicalPages and vmAllocatePhysicalPages)
• 9 in libpas/pas_page_malloc.c (madvise/mprotect in commit_impl and decommit_impl)
• 2 in libpas/pas_committed_pages_vector.c (mincore in pas_committed_pages_vector_construct)

Upstream Status

Apple's upstream WebKit has the identical zero-delay SYSCALL macro and has not addressed this. This fix is novel.

Related Issues

• Moving from Node to Bun spikes container CPU and memory usage until it crashes bun#17723 — Moving from Node to Bun spikes container CPU and memory usage (41 👍)
• Bun gets OOM-killed on trivial scripts bun#27371 — OOM on trivial scripts (bun -e "console.log('ok')")
• OOMKilled running Prisma migrations in Kubernetes - regression in 1.3.9 (works in 1.3.8) bun#27196 — OOMKilled on Prisma migrations (1.3.9 regression)
• Crash in bmalloc (pas_assertion_failed) on Windows x64 baseline in standalone executable bun#26982 — Crash in bmalloc after ~73 minutes
• tcmalloc making syscall madvise(MADV_DONTNEED) from a NOHZ_FULL cpu is causing the kernel to kick the CPU by raising a local timer interrupt google/tcmalloc#247 — madvise(MADV_DONTNEED) causing kernel interrupt storms
• runtime: MADV_HUGEPAGE causes stalls when allocating memory golang/go#61718 — MADV_HUGEPAGE causes 100+ ms stalls

Complementary Fix (Not in This PR)

MADV_DONTDUMP (the specific call that takes mmap_write_lock) could also be removed or made optional. It only affects core dump size, not allocation correctness. However, that's a behavioral change best evaluated separately.