Releases: sdogruyol/gcry
Release list
v0.18.0
Product release for upstream Crystal ≥ 1.21 — no compiler fork.
- Finalizer fix + Linux retain=0 → fat-app RSS ~1–1.6× (Linux); Darwin acik ~90% @ ~0.63×
- Darwin Kemal tip ~84%
/json@ ~1.01× - Linux Kemal headline still v0.16 carry (~87% @ ~0.80×)
- Stack maps included but dormant (
GCRY_PRECISE_STACKoff) - Opt-in
GCRY_TIGHT_GROW(not default)
Changed: Linux empty-chunk/large-cache retain default 0.
Escape:GCRY_EMPTY_CHUNK_RETAIN/GCRY_LARGE_CACHE.
See CHANGELOG.md.
v0.17.0
Darwin Kemal re-cut (first since v0.13) + Parallel TLAB-off + lazy sweep as a supported opt-in (~79% /json).
- macOS Kemal:
/json~84% @ ~0.93× RSS - Linux Kemal: carry v0.16 (~87% @ ~0.80×)
- Parallel opt-in: EC>1 + TLAB off + lazy (~79%); not default
- Fat-app: Linux ~90% / ~3.43×; Darwin ~71% / ~18×
- CI: hardened RSS leak gate (post-warmup sampling)
See CHANGELOG and docs/PERF-macos.md.
v0.16.0
EC1 thr recovery after Parallel-era STW / scrub / counter fallout. Supported
path remains EC parallelism 1, GCRY_TLAB off (Parallel+TLAB stays
experimental — FINDINGS only, not folded into PERF).
Performance
- Linux Kemal (same-host median-of-3,
wrk -c 100 -d 30, scrub on):/json
~87% of Boehm @ ~0.80× post-GC RSS;/~82% @ ~0.79×. Session
bench/log/linux/2026-08-01-093130/(cb4d7f2; idle/fromslash-recut/).
Fair Boehm ~40k baseline. See docs/PERF.md (Linux). - EC1 thr levers (Boehm ~40k fair): restore v0.15 parked-fiber scrub on EC1
(4 KiB blind clear; Parallel keeps 512 B +clear_range_safe). Tip with
512 B + safe retained ~4× morelive_objectsthan bebedae. EC1 alloc/free
counters use plain get/set (heap_counters_atomiconly when
EC_PARALLELISM>1) — avoid LOCK XADD/CAS on the hot path. - EC1 sweep pause: STW
live_objects/free_bytesupdates no longer
CAS-loop per dead object. Empty dormant/munmap freelist cleanup batches
into onerebuild_size_class_freelistper size class. Dormant post-STW
flush early-outs whendormant_chunk_bytes == 0. - EC>1 thr gap (experimental): auto-collect trylock-or-skip on
@post_stw(no waiter pile-up; wait_total ~11s/20s → ~0). Default major
threshold 64 MiB whenEC_PARALLELISM>1(GCRY_THRESHOLDstill wins;
EC1 stays 32 MiB). Same-host re-cut: gcry EC4/json~68% of Boehm EC4
@ ~53k abs (was ~52% @ ~36k). Long soak 100/100 soft=0 hard=0
(2026-07-31-ec4-soak-100-post-thr). NoPERF.mdfold-in. See FINDINGS. - Parallel empty-chunk reclaim opt-in: default stays off under EC>1 (thr).
GCRY_PARALLEL_DORMANT=1DONTNEEDs empty chunks (RSS ~3× better, thr ~25%
down on Kemal EC4).GCRY_PARALLEL_RELEASE=1adds munmap excess (hung in
A/B). EC1 dormant+munmap unchanged. See FINDINGS RSS A/B. - EC>1 alloc-path A/B:
GCRY_TLAB=1@ EC4 still ~½ of TLAB-off thr (soft 0
— keep opt-in).@alloc_lockaspthread_mutexdeadlocks under STW
(collections=0) — rejected; stay onCrystal::SpinLock. Fold
note_alloc_bytesinto the freelist lock (one acquire per small alloc /
TLAB hit). Session2026-07-31-ec4-alloc-thr-ab. NoPERF.mdfold-in.
See FINDINGS. - Atomic alloc counters:
bytes_since_gc/live_objects/free_bytes/
etc. areAtomicso TLAB hits need no@alloc_lockfor accounting. EC4
TLAB-off thr unchanged (~51k); TLAB-on still ~52% of off. Session
2026-07-31-ec4-atomic-counters. NoPERF.mdfold-in. See FINDINGS. - Per-size-class freelist SpinLocks: TLAB-off small alloc/free lock only
that size class (not global@alloc_lock). Large + TLAB table/refill keep
@alloc_lock(per-class refill hurt TLAB-on via@index_lock×find_block).
Quiet EC4/json~55k (was ~51k). Session
2026-07-31-ec4-sizeclass-locks. NoPERF.mdfold-in. See FINDINGS.
Fixed
-
EC1 STW stack scan thr regression (Parallel fallout): process-STW full
fiber/pthread scans added for EC>1 mid-swap were also applied on EC1
(main+SYSMON). Every Thread root fiber is named"main", so SYSMON hit a
full pthread map scan (phase_stacks0.02→3ms; Kemal/json86%→80%
Boehm). Restore cheap SP/stack_topother-thread scans when
!multi_mutator_threads?; keep aggressive Parallel path. Limit
foreign-SP scrub skip to Parallel only. Sessions2026-07-31-164302
(regress),2026-07-31-173530(fix); final cut above. -
Parallel
@suppress_collectrace: plainInt+=/-=under concurrent
realloclost decrements so suppress stuck high (≈4607) and auto-collect
never ran (collections=0, thr collapsed). UseAtomic(Int32). Exposed when
alloc counters left@alloc_lock(shorter critical section). See FINDINGS. -
chunk_containinglock during post-STW: skipped@index_lockwhenever
@collecting(not only@world_stopped). Flush keeps@collectingafter
start_world, so Parallel mutatorsindex_insertwhile peers realloc
unlocked → falseowns_user_pointer?(pointer is not a gcry allocationon
String::Builder). Lock skip only under true STW. Soft errors 0/60 after
empty-chunk gate (was 2–3/60). See FINDINGS. -
Parallel empty-chunk release off: under multi-mutator STW, skip empty-chunk
munmap even whenrelease_empty_chunksis on (EC1 unchanged). Residual
mark-miss × post-STW munmap surfaced as Kemal/jsonsoft
pointer is not a gcry allocation(22/40 → 3/40 with the gate; hard
deaths 0/40).GCRY_STW_STACK_LAGenv for LAG A/B (default 512 KiB). See
FINDINGS mark-miss triage. -
EC1
stw_sp_clampcounters: idle/stack_topother-thread scan now
incrementssp_clamp_fallbacks(missed after cheap-scan restore; aarch64 /
Darwin CIsamples/stw_sp_clampsaw hits=0 fallbacks=0). -
pattern_fuzzStride CI floor: raise Stride p99/max vs-baseline limit
20→80× after EC1 4 KiB parked-fiber scrub (quiet ~11×; GHA crystal-latest
hit ~45–57×). -
No live TLAB steal:
steal_from_other_tlabscould null another thread's freelist head while that thread was in lock-freetlab_alloc_small(TOCTOU dual-alloc). Removed cross-TLAB steal; idle freelists return via STWflush_all_tlabs.@tlab_stealsstays 0 (metric reserved for a future CAS steal). -
FREE-claim × minor: stack/thread FREE-claim cleared
FREEbefore the minor/old filter, so an old freelist node became USED-unmarked and scrub dropped it. Skip claim entirely for old nodes during minor (minor never munmaps old chunks); nursery nodes still claim+mark. -
Parallel worker STW stack scan:
scan_other_thread_stacksusedmax(stack_top, sp)for running fibers; stalestack_topabove hardware SP skipped live frames, so Parallel+TLAB in-flight mallocs were swept (pin saw FREE). Prefer suspend SP (+ x86_64 red zone), mark saved GP registers from the suspenducontext, and with TLAB scan the full fiber stack (SP/greg alone still flaked under Parallel>2). CI:stw_mt_property_test --tlab --nurserymixes minors. -
TLAB FREE-claim chain mark: stack/thread FREE-claim only marked the current freelist
user; TLAB batch tails reachable vianext_freestayed unmarked FREE, so empty-chunk release munmapped them andtlab_alloc_smallSEGVd inBlockHeader.free?(KemalGCRY_TLAB=1@ EC1). Claim now marks thenext_freechain (keep FREE on tails); abandon TLAB heads that failfind_block. -
Parallel mutator heap-index races (partial): under
EC_PARALLELISM>1,chunk_containing/ last-chunk cache racedindex_insert(falseowns_user_pointer?/ corruption). Added@index_lock;with_alloc_lockalways locks (was a no-op when TLAB off);ensure_tlabsboots under@alloc_lock. Process-STW other-thread fiber stacks always full-scan. KemalEC>1HTTP still fails — see FINDINGS. -
TLAB per-slot freelist locks: Parallel dual-alloc on lock-free TLAB heads (
ec_alloc_stressdouble-free /not a gcry allocation). Per-slotCrystal::SpinLock(StaticArray — no GC malloc under@alloc_lockat boot). STWflush_all_tlabsmust not take slot locks (suspended mutator may hold them). Refill always re-claims under the slot lock. KemalEC>1still open. -
STW running-fiber scan:
scan_all_fiber_rootsskippedfiber.running?, relying onthread.@current_fiber; under Parallel that TLS can be briefly nil so stacks were missed. Under process STW, scan running fiber stacks too; ifcurrent_fiberis nil, fall back to pthread stack bounds + greg. -
STW × ExecutionContext deadlock (
GCRY_STRESS): signal-suspendingSYSMONdeadlocks (fiberyieldwait, or lostSIG_RESUMEleavingsigsuspendforever). Fix: skip SIGPWR for the Monitor; cooperative STW via@world_stoppedbarriers inallocate/lock_read; busy-wait@suspendedfor other threads (noyield_current); holdThread.lockfor stop→start; harden resume handshake; forbid process collect onSYSMONso the Monitor cannot STW-suspend the mutator. -
TLAB@EC1 measured: correctness OK (Kemal 20/20 default + thr=32KiB; STW MT
--tlab)./jsonthr ~71–77% of TLAB-off on same host — keep opt-in (GCRY_TLAB=1), not an EC1 default. Hit-pathfind_blockdominates; stripping it SEGVs. See FINDINGS. -
EC>1 thr vs Boehm (measured): Kemal EC4 TLAB-off
/json~23% of Boehm EC4 and ~0.52× gcry EC1 (session2026-07-31-100844-ec-parallel-thr). Correctness quieter; Parallel still anti-scales — experimental. -
Multi-mutator STW stack LAG: full
guard→bottomon every parked fiber dominated EC4phase_roots(~100ms+/collect). Prefer suspend SP−red_zone when present; otherwise scan fromstack_top − 512KiB(not full guard). Same-host A/B/jsonmedian-of-5: LAG ~30k vs stw_full ~16k (~1.9×); EC4 soak 30×8s 0/30. Quiet re-cut vs Boehm: EC4/json~37% Boehm EC4 and ~0.87× gcry EC1 (was ~23% / ~0.52×).GCRY_TLAB=1@ EC4: soak 3/20, thr not above good TLAB-off — keep opt-in. See FINDINGS. -
EC4 post-STW queue: SpinLock wait on
@post_stwburned ~8–11s/20s of worker time under Parallel HTTP. Switch to embeddedpthread_mutex; auto-collect coalesce when a peer already cleared the debt; pause stats exclude queue wait. EC4/json~40k med (d=20) + soak 20/20 (was ~22k + crash outliers). Quietd=30re-cut vs Boehm: EC4/json~52% Boehm EC4 and ~1.17× gcry EC1 (was ~23% / ~0.52× pre-LAG). Long soak 96/100 (4× SEGV/MARK_MISS). See FINDINGS. -
Post-STW flush keeps
@collecting+@suppress_collect: clearing@collectingbefore flush allowed stress/auto re-entry while still holding@post_stw_lock(non-recursive SpinLock). Hold collecting through flush. -
realloc suppress-collect + Boehm-like thread stacks: growing
reallocsets@suppress_collectaround the fresh allocate so a mark miss cannot free-the...
v0.15.0
Correctness release: process-STW × TLAB freelist UAF class fixed and CI-gated;
process-STW MT property harness; acikturkiye Linux re-cut measured; shard RSS
dead-end defaults documented. Supported path remains EC parallelism 1,
GCRY_TLAB off (Parallel+TLAB stays experimental).
Performance
- Linux Kemal (same-host median-of-3,
wrk -c 100 -d 30, scrub on):/json~86% of Boehm @ ~0.77× post-GC RSS;/~86% @ ~0.76×. Sessionbench/log/linux/2026-07-29-151144/(bebedae). Collector defaults unchanged vs 0.14 — thr within host noise of the v0.14 ~89% cut. See docs/PERF.md (Linux). - acikturkiye Linux re-cut (measured):
/api/v1/~90% of Boehm thr @ ~2.54× post-GC RSS (median-of-3,wrk -c 100 -d 30, scrub on). Sessionbench/log/linux/2026-07-29-112202/(9decd01). Replaces the v0.14.0 ~93% / ~2.65× estimate. See docs/ACIKTURKIYE.md. - Shard RSS A/B (defaults unchanged): same-host cuts rejected as defaults — Linux HOLED
GCRY_PAGE_DONTNEED, process-default curatedHTTP::Headers::KeyHash layout, collect-time mutatorclear_stack, Linux 1 MiB large-cache floor. Keep fiber scrub, Linux 4 MiB large-cache, HOLED opt-in; Headers layout stays app-side /GCRY_AUTO_LAYOUTS. See docs/ACIKTURKIYE.md “Don’t bother”.
Fixed
- Explicit-root list × process STW race:
add_root/delete_rootcould run concurrently withstop_world, freezing a mutator mid-list splice so@roots.eachwalked a freed/next-corruptRootNode(SEGV atrun_collectionduringstw_mt_property_test). Serialize mutations with@roots_lockacquired before STW; collector may mutate without the lock while@world_stopped. - Parked-fiber scrub on thinly mapped stacks: Cap wipe to the same 512 B fiber path as
clear_stackand zero only readable pages viaRoots.clear_range_safe(defense in depth; Crystal fiber stacks grow on demand). - TLAB + Parallel under process STW: mid-
tlab_alloc_smallSTW could leave FREE freelist nodes only reachable from mutator stacks; mark ignored FREE, then empty-chunk release munmapped them (andunlink_freelist_rangecould coerce USED→FREE). Fix: claim FREE stack/thread roots when TLAB+STW (clear FREE but keepnext_freeso scrub can walk the chain —set_usedwas severing freelists → OOM), freelist scrub after flush/mark (TLAB-only), flush only FREE nodes, TLAB epoch + detach-before-claim (no dual-alloc after flush), no nestedcollectunder@alloc_lock(deadlock), unlock-and-collect retry on refill miss, steal stranded TLAB freelists, skip nilThread#current_fiberunder Parallel. CI gatesstw_mt_property_test --tlab --workers=2,4.
Added
- Process-GC STW MT property harness:
bench/stw_mt_property_test.cr(-Dgc_none) runs Parallel allocator workers while the default EC pins roots (ACK handshake) andGC.collects under real STW. Closes the gap left by library-heapmt_property_test(stop_the_world=false). CI gates--workers=2,4and--tlab --workers=2,4. (make stw-mt-property-test)
Changed
- Docs / knobs: Linux HOLED page release documented as opt-in (post-STW; not “STW-heavy”). Large-cache defaults clarified (Linux process 4 MiB, Darwin 1 MiB). Darwin
GCRY_DISABLE_PAGE_RELEASE=1/GCRY_DISABLE_MADVISE=1explicitly clearmadvise_free_pages.
v0.14.0
Trust and tooling release: industry-style test suite, debug observability, and a
measured Linux Kemal re-cut. Collector throughput unchanged; Kemal post-GC RSS
now measured (not estimated).
Performance
- Linux Kemal (same-host median-of-3,
wrk -c 100 -d 30, scrub on):/json~89% of Boehm @ ~0.79× post-GC RSS;/~89% @ ~0.78×. Sessionbench/log/linux/2026-07-29-035426/. See docs/PERF.md (Linux). Fat-app (acikturkiye) not re-cut — still ~93% thr / ~2.65× RSS est. (ACIKTURKIYE.md).
Added
- Debug invariant checker (
GCRY_DEBUG_INVARIANTS=1): validates heap invariants at runtime --live_objectscounter accuracy, freelist cycle/consistency checks, chunk index integrity, and block overlap detection. Hooks intomalloc,free, andcollect. Diagnostics usewrite(2)/ no managed-heap alloc (not a claim that GC is async-signal-safe).-Dgcry_invariant_abortfor core dumps. ExposedHeap#each_chunk,#freelist_for,#nursery_freelist_forfor the checker. CI runs invariants on every PR. (spec/invariant_spec.cr,make invariants, CIDebug invariantsstep.) - Coverage infrastructure:
spec/all_specs.crentrypoint for kcov (DWARF-based line/branch coverage).ci/coverage.shwrapper runs kcov +crystal tool unreachable+crystal tool macro_code_coverage.make coverage/coverage-kcov/coverage-unreachable/coverage-macrotargets. CIcoveragejob builds the spec binary, installs kcov from Debian, and uploads the report. (ci/coverage.sh,Makefile,.github/workflows/ci.yml) - Memory safety CI:
make asanbuilds and runs specs with AddressSanitizer (-Dasan).make valgrind-samplesruns samples under Valgrind memcheck (--leak-check=full). CIasanandvalgrindjobs on every PR. (Makefile,.github/workflows/ci.yml) - Deterministic replay fuzzing:
bench/fuzz.crrewritten with--seed=,--seconds=,--log=, and--replay=flags. Fuzz logs every operation to a replayable log file (opcode + args). Replay mode reads the log and replays the exact sequence of heap operations. Op 9 (spawn + Channel) excluded from logs as non-deterministic Crystal runtime. CI runs fuzz + replay on every PR. (make fuzz-replay FUZZ_LOG=path, CIFuzz with log + replaystep.) - Property-based testing:
bench/property_test.cr-- random alloc/free/collect sequences with deep heap invariant verification:live_objectscounter accuracy (reported == walked count),heap_size== sum of chunkmapped_bytes, freelist consistency, and per-nodelive?assertion. 100k iterations in ~8s. (make property-test, CIProperty teststep.) - Layout property test:
bench/layout_property_test.cr-- 5 self-contained sub-tests verifying precise scan offset correctness, conservative fallback, leaf layout (scan_cap=0), noscan offset keep-alive semantics, and scan_cap limiting. Runs 10k iterations in ~2.5s. (make layout-property-test, CILayout property teststep.) - MT property test:
bench/mt_property_test.cr-- concurrent allocation via fiber workers (2, 4, 8) with periodic collect; verifies no objects lost under concurrent alloc,live_objectscounter accuracy after TLAB flush, and parallel mark (workers=2) produces the same live set as serial mark (workers=1). 500 iterations × 3 worker counts in ~2.4s. (make mt-property-test, CIMT property teststep.) - 24-hour soak test:
bench/soak.cr-- sustained load with alloc storm (~1000 obj/s), periodic collect (1 Hz), fiber spawn (10 Hz), finalizer load (100 obj/s), WeakRef via disappearing links (10 Hz). Hourly telemetry: heap size, free bytes, live objects, pause p50/p99, RSS. Post-soak RSS check (< 10% growth) and drain verification. Weekly CI cron (Monday 06:00 UTC). (make soak, CIsoakjob.) - Alloc pattern fuzzing:
bench/pattern_fuzz.cr-- 3 allocation distributions (Zipfian power-law, bimodal small+large, stride array-growth) each checked against baseline uniform-random. Verifies pause p99 < 8-10x baseline and RSS growth < 10%. 200 phases × 5000 objects per phase. (make pattern-fuzz, CIAlloc pattern fuzzstep.) - Thread storm test:
bench/thread_storm.cr-- 3 phases: thread spawn storm (OS threads doing alloc/free/collect in batches), rapid thread create/destroy (250 short-lived threads), CrystalSignal.trapdeferred alloc (event-loop mutator path; GC is not async-signal-safe — see POLICY.md). 1000+ iterations total, 0 errors. (make thread-storm, CIThread stormstep.) - OOM scenarios:
bench/oom_test.cr-- 3 phases: bounded heap (low gc_threshold, 500 iterations, no crash), mmap failure (graceful OutOfMemoryError), finalizer under OOM (no crash under pressure). (make oom-test, CIOOM teststep.) - Bug-fix test policy:
CONTRIBUTING.mdwith "bug fix must include test" rule,.github/PULL_REQUEST_TEMPLATE.mdwith reproducing test checkbox, andspec/regression/directory with 4 regression tests (live_objects dormant chunk, hash_layout entries_size, scan_cap alloc_size mismatch, signal_stack false root). (spec/regression/, CI regression jobs.) - API misuse test suite:
spec/api_misuse_spec.cr-- tests coveringGC.free(null),GC.realloc(null, 0),GC.malloc(0),GC.malloc_atomic(0),Gcry.add_root(null),Gcry.register_disappearing_link(null, ...),collectinside finalizer (no deadlock), CrystalSignal.trapdeferred alloc (Linux; not async-signal-safe),add_rootwith large pointer, alternating malloc/free. (make spec, CIspecstep.) - Fork reinit test:
bench/fork_reinit.cr-- standaloneLibC.fork+after_fork_child_reinit+ alloc in child + parent continues allocating after collect. 3 assertions, all pass. (make fork-test, CIFork reinit teststep.) - Finalizer complex scenarios:
bench/finalizer_complex.cr-- 7 phases: finalizer chain, finalizer callingGC.collect, finalizer adding root (resurrection), finalizer + disappearing links interaction, finalizer under heavy allocation pressure (500 objects), finalizer creating 1000 objects, and many disappearing links (200). 8/8 assertions pass. (make finalizer-complex, CIFinalizer complex scenariosstep.) - Perf regression alerting:
bench/perf_smoke.shrewritten with variance protocol -- 5 wrk runs per path, min/max discarded, median reported, noise ratio computed (IQR/median). same-host variance protocol (N wrk runs, min/max discard, median, noise ratio); gate is gcry /json % of Boehm only. Absolute RPS is not compared across hosts. Per-run JSON underbench/log/uploaded as CI artifact. (bench/perf_smoke.sh, CIperf smokejob.) - Microbenchmark suite:
bench/micro/run_all.cr-- 6-phase suite measuring alloc latency (10 size classes, p50/p99/max), free latency, collect latency (5000 obj, p50/p99/max), TLAB refill cost, STW suspend/resume latency, and GC lock overhead. Runs in < 10s. (make microbench, CIMicrobenchmark suitestep.) - Pause time budget:
bench/pause_budget.cr-- major p99/max budgets scaled to live set, incrementalcollect_a_littleslice budget (STW-aware), minor vs major pause ratio. (make pause-budget, CIPause budgetstep.) - RSS leak detection:
bench/rss_leak.cr-- cyclic alloc/free/collect; gate is intra-run RSS growth only (late-half vs early-half <10%). RSS/heap ratio is informational. Writes gitignoredbench/trend.json. (make rss-leak, CIRSS leak detectionstep.) - Darwin platform parity tests (Phase 6.1):
spec/platform_darwin_spec.crasserts soft-dirty/mprotect stubs return unsupported,pthread_get_stackaddr_npstack bounds contain the current SP, and host-page-alignedMADV_FREE_REUSABLEreclaim works.process_specDarwin section exercises Machthread_suspend/resumeSTW round-trip + SP clamp under-Dgc_none. Windows process-GC gap documented indocs/INTEGRATION.md(crystal#15173 HeapAlloc stub ≠ gcry port). - Compiler GC contract (Phase 6.3):
bench/compiler_gc_contract.crmirrors Crystalspec/std/gc_spec.cr(stats/prof_stats/enable) plus malloc/realloc/collect, disable/enable, and runtime@crystal_type_idvscrystal_instance_type_id. CI also runscrystal tool hierarchy/unreachableon gcry sources. (make compiler-gc-contract) - Kemal E2E (Phase 6.4):
bench/kemal_e2e.shhits every endpoint (/,/json,/gc-collect,/gc-stats,/metrics) before and after concurrent wrk load. CI runs 60s; full 10-min DoD viaKEMAL_E2E_DURATION=600 make kemal-e2e. - GC trace log (Phase 7.1):
GCRY_TRACE=1emits NDJSON events (alloc/freesampled,collect_start/collect_end,finalizer,barrier_arm) viaGcry::Trace. Reentrancy guard avoids malloc recursion. (make trace-smoke,spec/trace_dump_spec.cr) - Heap dump (Phase 7.2):
Gcry.dump_heap(io)/dump_heap_addresses/heap_dump_gone/newfor live-object NDJSON and leak diffs. Dump count matcheslive_objects. - Mutation harness (Phase 7.3):
bench/mutations/run.sh— 10 hand-crafted sed mutants; kill suite scores 10/10. Feasibility notes indocs/MUTATION.md.
Fixed
-
Gcry::Traceunder-Dgc_none: do notrequire "json"or write via abstractIO— both pulled JSON/OpenSSL into the GC bootstrap and broke process builds. Trace now emits NDJSON with a stack buffer +LibC.writeto a raw fd. -
Darwin
release_physical_pagesspec: do not assert immediate zero-fill afterMADV_FREE_REUSABLE(kernel may keep contents until reclaim). Assert aligned success + still-mapped only. -
Nursery HTTP::Headers regression: moved from
process_specto standalonebench/nursery_headers.cr— Spec + process GC + nursery was flaky on CI (SEGV during Spec reporting). -
Process parallel mark: moved from
process_spectobench/parallel_mark_process.crfor the same Spec+process-GC flake; CI retriesprocess_specup to 3 times. -
**
live_objectscounter drif...
v0.13.0
Changed
- Darwin
empty_chunk_retain8 MB → 512 KB: AggressiveMADV_FREE_REUSABLEreclaim on Darwin. Kemal RSS drops from ~160 MiB to ~18 MiB (1.04× Boehm). ACIKTURKIYE RSS unchanged (~700 MiB); conservative live set remains the dominant driver. scrub_fibers_enabled= true (Linux + macOS): Default-on fiber stack scrubbing to reduce false roots from parked fiber stacks. Linux: Kemal RSS 0.99×→0.95×, acikturkiye RSS 3.00×→2.65×. macOS: ACIKTURKIYE RSS steady at ~700 MiB (conservative live set dominant). Opt-out viaGCRY_DISABLE_SCRUB_FIBERS=1.- Darwin
gc_threshold32 MB → 16 MB: More frequent major collections on Darwin; pause halved (47→25 ms p50) on ACIKTURKIYE. - Darwin
small_chunk_bytes128 KiB → 256 KiB: The 128 KiB chunk inflated collection count (~290 majors in 30s) and crushed acikturkiye throughput to ~57% Boehm. 256 KiB recovers throughput to ~78% without meaningful Kemal RSS cost (1.06× vs 0.88×). Set ingc_override.crfor Darwin only; library default stays 128 KiB. Escape:GCRY_CHUNK_BYTES=131072.
Added
- Darwin large-freelist
MADV_FREE_REUSABLE:darwin_release_large_freelist_pagesissuesMADV_FREE_REUSABLEfor every cached large-object chunk after major collection on Darwin, dropping physical pages without unmapping. Linux unchanged (mmap-resident for cache budget).
Performance
- macOS v0.13.0 (Apple Silicon M2 Pro, median-of-3,
wrk -c 100 -d 30,--release, 256 KiB chunk default):- Kemal:
/92.6% of Boehm;/json83.9%; post-GC RSS 0.93–1.06×. - ACIKTURKIYE
/api/v1/: 77.9% of Boehm, post-GC RSS 15.8× (~600 MiB). 0 crashes across 3 trials. - See docs/PERF-macos.md, docs/ACIKTURKIYE-macos.md.
- Kemal:
v0.12.0
Added
-Dgcry_side_bitmap(opt-in): sideMarkBitmapmmap path kept for experiments. Default is in-headerMARKagain after Linux A/B showed bitmap default at 82%/json@ ~9.2× RSS vs header 89% @ 0.99× (acikturkiye 50%→93%, 5.6×→3.0×) —bench/log/bitmap-ab/FINDINGS.txt.- Bitmap shrinking + adaptive headroom (P1.1):
MarkBitmap#shrink_to_fit!reduces the side-mark bitmap mmap when the heap range contracts. Adaptive headroom (25% of recent growth history) prevents immediate re-growth. Combined with tighterupdate_heap_bounds_after_unmap, Kemal RSS drops from ~10× to ~5–7× (when-Dgcry_side_bitmap). - Darwin
MADV_FREE_REUSABLE(P1.1, macOS):release_physical_pagesswitched from the expensive 3-syscallmach_vm_deallocate+allocate+protectto a singlemadvise(..., 5).empty_chunk_retainlowered from 64 MiB to 8 MiB on Darwin (no cost;MADV_FREE_REUSABLEis cheaper than the retain budget). - Deferred madvise — STW pause damping (P1.4): All
madvise/ page-release syscalls defer to post-STW flush functions (flush_pending_dormant_chunks,flush_pending_page_release_chunks). DORMANT/HOLED flags set during STW; actual syscalls run after threads resume, eliminating kernel VM lock contention that caused 132–150 ms pause tails. - Cross-chunk dormant coalescing (P1.4):
flush_pending_dormant_chunksmerges contiguous dormant chunks into a singlemadviseregion (one syscall per run instead of one per chunk). - Per-chunk free-page coalescing (P1.4):
dontneed_free_pages_in_chunkpre-computes a live-page mask and issues onemadviseper contiguous free run instead of one per free page (reduces from up to 64 syscalls/chunk to 1–3). - Auto-layouts (P2.1):
Gcry.register_layoutswhole-program walk +@unsafe_layoutsblacklist (Cry/Crystal::*/LibC::*; metriclayout_unsafe_skips). Opt-in viaGCRY_AUTO_LAYOUTS=1(Linux Kemal/json~−7pp vs builtins-only — seebench/log/thr-abis). Escape when opted in:GCRY_DISABLE_AUTO_LAYOUTS=1. - Per-source root reject counters: New
type_id_stack_rejects/type_id_static_rejects/type_id_thread_rejectscount where false roots come from (fiber/mutator stacks, BSS/data, TLS). Plustype_id_root_false_negativesis now exposed in/gc-stats, metrics, and Prometheus — was tracked but never surfaced. Sum invariant:stack + static + thread == type_id_root_rejects. - Adaptive nursery threshold:
@nursery_thresholdadjusts dynamically after each minor based on the moving-average survival rate (last 10 minors). Target survival rate is 50%; when survival rises above it the threshold grows by 25% per minor (reducing collection frequency); when survival drops below 25% the threshold shrinks by 25% (collecting sooner to limit survivor pressure). Clamped to [64 KiB, 8 MiB]. Default-on for process GC (adaptive_nursery=true); disable viaGCRY_DISABLE_ADAPTIVE_NURSERY=1. - Large-cache LRU eviction + adaptive retain (P3.3):
cache_large_chunkinserts at tail (LRU).trim_large_cacheevicts from head. Adaptive retain: after each major, hit-rate above 50% doubles retain (capped at 64 MiB); hit-rate below 10% halves it (floor 1 MiB). Default: 1 MiB on Darwin (macOS), 8 MiB on Linux. - Bitmap headroom reduced 25% → 12.5%:
note_bitmap_growthnow usesavg_range >> 3instead of>> 2, shrinking side-mark bitmap reserve — less RSS waste on stable heaps.
Fixed
- Hash layout walk used
entries_capacityinstead of Crystalentries_size: precisescan_hash_objectiterated(1 << indices_size_pow2) / 2slots. Afterrealloc, slots past@size + @deleted_countare uninitialized; non-zero garbage@hashwords caused false marks / mutator UAF under acikturkiye (GCRY_DISABLE_LAYOUT=1was the only green bisect). Now walks@size + @deleted_count, capped by capacity. Also word-scans@block(Proc?, 16 bytes) instead of treating it as a single pointer. - Layout
scan_caprequiredalloc_sizematch: on size mismatch (raw buffer whose leadingInt32collided with a registeredtype_id), the old path still applied that type'sscan_capand returned — truncating the mark scan and dropping live pointers (acikturkiye SEGV with layouts on; green withGCRY_DISABLE_LAYOUT=1). Size mismatch now falls through to full conservative scan.
Changed
- In-header MARK is default again: side mark bitmap moved to
-Dgcry_side_bitmapafter Linux HTTP A/B (bench/log/bitmap-ab). Headline cut: Kemal/json88.8% @ 0.99× RSS; acikturkiye 92.8% @ 3.0× — docs/PERF.md, docs/ACIKTURKIYE.md. - Nursery + incremental default-off for process GC: Linux no longer enables
nursery/incremental_autoby default. Soft-dirty false-negatives under WSL release HTTP (Kemal) caused Hash key UAF / SEGV (0x0/0x4/0x11). Opt in withGCRY_NURSERY=1/GCRY_INCREMENTAL=1after measuring. Darwin unchanged (already off). Related fixes kept:reallocpins old buffers across collect; explicit roots skiptype_id_gate; old→young always full-walks (soft-dirty is additive only) with one-level buffer chase. incremental_autodefaults (P1.3, Linux/Darwin): (superseded — both off by default; see above.)GCRY_AUTO_LAYOUTSopt-in (P2.1): briefly default-on; reverted after Linux A/B — builtins-only/json~85% Boehm vs auto-on ~78% (bench/log/thr-abis). SetGCRY_AUTO_LAYOUTS=1to enable.- Bench default build:
bench/run_all.shuses pure--releaseagain (PERF.md).--release --debug --error-tracecost ~15–18pp thr; useCRYSTAL_FLAGS/DEBUG=1only for SEGV hunting. - Nursery default-on for Linux process GC: (superseded — off by default again; see above.)
- Darwin blacklist re-enabled: Previously default-off on Darwin (freelist-abandonment spiral under all-conservative scanning). Layout-precise scans (P2.1) cut false root hits sharply, making the blacklist safe. Escape via
GCRY_DISABLE_BLACKLIST=1. - Darwin aggressive free-page release:
flush_pending_page_release_chunkswalks ALL kept size-class chunks (not just HOLED) on Darwin.MADV_FREE_REUSABLEis page-table-level (no VM lock churn), so the extra walk is cheap per major. - Darwin large cache reduced to 1 MiB (adaptive): Adaptive LRU policy starts at 1 MiB on Darwin (vs 8 MiB on Linux). mach_vm reclaim already punches holes on free, so a fat cache is wasteful; 1 MiB floor avoids mmap churn for the common case.
Performance
- Linux Kemal (WSL2 x86_64, median of 3, pure
--release, in-header MARK default, sessionbench/log/2026-07-26-173602/):/90.4% of Boehm;/json88.8%; post-GC RSS 0.99×. acikturkiye/api/v1/: 92.8% of Boehm, post-GC RSS 3.00×. Side-bitmap A/B (2026-07-26-171942):/json82.3% @ ~9.2×, acik 50.1% @ 5.58×. See docs/PERF.md, docs/ACIKTURKIYE.md. - macOS Kemal (Apple Silicon M2 Pro, median of 3, pure
--release, in-header MARK default, sessionbench/log/2026-07-26-181318/):/85.4% of Boehm;/json86.5%; post-GC RSS 1.34–1.36×. acikturkiye/api/v1/: 76.7% of Boehm, post-GC RSS 22.3× (RSS improved 2.6× vs prior session; conservative live set remains the dominant driver). See docs/PERF-macos.md, docs/ACIKTURKIYE-macos.md. - STW pause tail eliminated: deferred madvise removes kernel VM lock from the STW window. Max pause drops from 132–150 ms to well under 50 ms on Kemal
/jsonc=100.
v0.11.0
Added
- Side mark bitmap: mark bits live in a separate mmap (one bit per word-aligned heap address), replacing the in-header
MARKflag.clear_all_marksis now aUInt64word-by-word zero over the bitmap (full memory bandwidth) instead of a per-block header write.marked?/set_mark/clear_markare answered from heap-inlined mirror fields (@mark_bitmap_base/@mark_bitmap_base_addr/@mark_bitmap_cap_bits) so the mark hot path no longer dereferencesGcry.current_mark_bitmapplus aMarkBitmapmethod. Bitmap relocation publishes the new base pointer before unmapping the old mapping;Heap#destroyclears the global first then nulls the mirrored fields so stale readers short out. - Chunk coalescing on flush:
flush_pending_empty_chunkswalks the pending list and merges fully-contiguous chunks (next.base == current end) into singlemunmapregions (one syscall + one VMA teardown per run instead of one per chunk). Stricter than the naive<=check so chunks with a gap (kernel-placed VMA between) are flushed independently. empty_chunk_retainbumped to 64 MiB in the process GC override — keeps recently-freed chunks asMADV_DONTNEEDdormant (kernel drops the physical pages, VMA cache survives for fast reuse). 0 MiB regressed ~70% via mmap/madvise cycling; 32 MiB regressed ~50% (reclaim thrashing); 64 MiB is the sweet spot.
Changed
- HDR pause histogram:
@pause_hdris aStaticArray(UInt64, 64)with bucket indices chosen byclzon the elapsed-ns value (1–3 ns, 4–7 ns, …). Exposed viaGcry.pause_percentile_hdr_ns(p)andGcry.pause_hdr_snapshot(per Kemal/gc-stats). type_idgate instrumentation:type_id_root_false_negativescounter for objects rejected by the ambient-root gate that later proved live by other means; bounds the false-negative rate under workloads that mix static-root scanning with type_id gating.- Mark-stack prefetch + chunk batching: the mark loop walks chunk ranges in size-class order with
__builtin_prefetchon the next chunk header; cache miss count drops on Kemal/json.
Fixed
- Flush coalescing under-counted
unmapped_byteson Linux. The old<=coalescing predicate (nxt.base <= run_end) silently skipped chunks whose ranges overlapped or had a small gap (4 KiB page between two separately-mmap'd size-class chunks is common on Linux x86_64). The result wasunmapped_bytes~½×released_chunk_bytesonspec/collect_spec.cr:159("munmaps fully free size-class chunks on major"), failing CI on Linux x86_64 + aarch64 native + aarch64 cross-compile. Tightened tonxt.base == run_end(only fully-contiguous chunks coalesce) so the release count and the unmapped count always match. Verified incrystallang/crystal:1.21.0Docker (Linux x86_64): 94/94 unit specs + 13/13 process specs + 5 samples + format + Ameba all pass.
Performance
- macOS Kemal (Apple Silicon, median of 3, scrub off):
/~100% of Boehm (was ~97%);/json~94% of Boehm (was ~90%); post-GC RSS ~10× (was ~0.97× — see notes). Latency p50:/json2.3 ms (was 18 ms, −87%);/1.7 ms (was 14 ms, −95%). p99 latency within 2× of Boehm on both paths. See docs/PERF-macos.md. - Note on RSS: the side mark bitmap itself allocates a separate mmap region covering the live heap (1 bit per word-aligned address). For the Kemal workload this adds ~200 MiB of mapped address space on top of the managed heap — hence the ~10× post-GC RSS. This is the explicit price paid for moving mark bits off the object headers; further reduction requires the bitmap to follow heap-range tightening (see
ensure_bitmap_covers) or a shared page-cache strategy. The throughput + latency win more than compensates for the higher mapped set on the HTTP workload. - Linux numbers unchanged (this host is Darwin) — re-record on Linux before citing a new Linux cut. See docs/PERF.md.
v0.10.0
Added
- macOS process GC (the headline):
-Dgc_none+require "gcry"is a real collector on Darwin (arm64 + x86_64), Crystal ≥ 1.21 — not stubs.- STW: Mach
thread_suspend/thread_resume(signal STW under HTTP was ~hang / ~2 req/s) - SP clamp:
thread_get_state+pthread_get_stackaddr_npstack bounds - Static roots: dyld main-image
__DATA/__DATA_CONST(__data/__bss/__common; skip__const) - Free-page RSS: host-page
mach_vm_deallocate+allocate(FIXED)(Apple Silicon 16 KiB;MADV_DONTNEEDdoes not drop Darwin RSS) - Defaults: page blacklist off (opt-in
GCRY_BLACKLIST=1);large_cache_retain0 - CI:
macos-latestnative specs + samples
- STW: Mach
Gcry.register_set(T)— registersHash(T, Nil)forSetbacking maps.GCRY_SCAN_CAPS=1— optional whole-programinstance_sizeofscan caps (fat-app live set often unchanged).
Changed
- Layout builtins: broader curated coverage — primitive/
Stringarrays,Set-backing hashes,Hash/Array+JSON::Any,IO::Memory(noscan buffer), moreDeques. Still not whole-programGCRY_AUTO_LAYOUTS. - Layout correctness:
Pointer(T)noscan uses!T.has_inner_pointers?(safe forArray(JSON::Any)). Hash keys/values with inner pointers word-scanned. - Mark: size-class mismatch falls back to
scan_capwhen present; precise entries storeinstance_sizeof. - Large objects: mmap aligned to
Platform.host_page_size;LARGE_CACHE_LIMIThard-caps freelist retain. - Blacklist: page granularity uses
host_page_size. - Docs: Linux vs Darwin PERF / ACIKTURKIYE split; README highlights macOS.
Performance
- macOS Kemal (0.10.0 cut, Apple Silicon, median of 3, scrub off):
/~97% of Boehm;/json~90%; post-GC RSS ~0.96–0.97× — see docs/PERF-macos.md. - macOS acikturkiye
/api/v1/(median of 3): thr trial-median ~80%; post-GC RSS ~11.8× (dense conservative-live; reclaim works) — see docs/ACIKTURKIYE-macos.md. - Linux Kemal / acikturkiye cut numbers unchanged from 0.9.0 (this host is Darwin; re-record on Linux before citing a new Linux cut) — docs/PERF.md, docs/ACIKTURKIYE.md.
v0.9.0
Added
- Process-GC parallel mark (STW-exempt): with
GCRY_PARALLEL_MARK=N/parallel_mark_workers > 1, helpers are rawLibC.pthread_createthreads (not Crystal::Thread), sostop_worlddoes not suspend them. They steal grey objects under@mark_lock(parallel_mark_stolen). Fork child abandons the pool viareset_mark_workers_after_fork. - Library-heap parallel mark: with
parallel_mark_workers > 1andstop_the_world == false, helperThreads steal grey objects (parallel_mark_stolen). - Stack scrubbing (no Crystal patch):
GCRY_CLEAR_STACK=1zeros a window below SP (skips x86_64 red zone; default every 16 allocs) without calling Fiber/Thread APIs;GCRY_SCRUB_FIBERS=1zeros a capped window below each parked fiber's saved SP before mark (not the full unused stack — that faults pages in and blows RSS). Metrics:clear_stack_*/fiber_scrub_*(json_stats + Prometheus). Not stack maps; measure before enabling as default. - Richer
Gcry::Observability.json_stats(phase timers, mapped/live bytes, TLAB, parallel-mark, barrier) — Kemal/gc-statsuses it. - Prometheus: TLAB, parallel-mark, phase, layout, SP clamp, barrier, size-class live / released chunk gauges;
gcry_clear_stack_*/gcry_fiber_scrub_*. - Median-of-3 helpers:
bench/median_kemal_boehm.sh,bench/median_acikturkiye_boehm.sh.
Changed
- README / HARDENING / POLICY:
GCRY_PARALLEL_MARKis real for process GC (pthread steals), not counter-only — and labeled experimental / measure first (Kemal/json+ acikturkiye/api/v1/thr regressed vsN=1in same-host wrk). - README / HARDENING: document
GCRY_DISABLE_*escapes,GCRY_TLAB, stack-scrub knobs. - Dogfood docs: docs/ACIKTURKIYE.md + docs/API.md point at Observability routes; acikturkiye
make run-demo-gcry/ README GC section. - Same-host Kemal (0.9.0 cut, median of 3, scrub off):
/~89% of Boehm;/json~92%; post-GC RSS ~0.97× — see docs/PERF.md. - Same-host acikturkiye
/api/v1/(median of 3, scrub off): thr trial-median ~93%; post-GC RSS ~2.84× (was ~3.20× at 0.8.0) — see docs/ACIKTURKIYE.md.
Fixed
clear_stackaarch64 SEGV: wipe used approximatepointerof(local)as SP (mid-frame). With no x86_64 red zone that zeroed the leaf frame (Invalid memory access @ 0x0on CItest (aarch64 native)). Now reads hardware SP (Roots.hardware_stack_pointer) plus a leaf margin.