Fuzz (nightly) scheduler_solver has failed 97/97 runs since at least 2026-04-25 — ASan allocator amplification, not a solver defect

[avrabe]

Summary

Fuzz (nightly) has failed 60 out of its last 60 runs — every run back to its oldest
recorded one, 2026-06-01. It has, as far as the run history shows, never passed.

$ gh run list --workflow "Fuzz (nightly)" --limit 60 --json conclusion \
    --jq '[.[]|.conclusion]|group_by(.)|map({(.[0]):length})'
[{"failure":60}]

Trace-Topology Fixture Generation is in the identical state and is filed separately.

Nothing surfaced this, because neither workflow is a required context — so a
100%-failure gate is indistinguishable on the PR page from a gate that is working. Sixty
consecutive reds produced zero signal. That is the part worth fixing beyond the immediate
cause: a gate nobody reads is not a gate.

The failure

fuzz_scheduler_solver (1h) OOMs about 10 minutes in. The other two targets
(fuzz_aadl_parse, fuzz_codegen_roundtrip) pass.

==5922== ERROR: libFuzzer: out-of-memory (used: 2056Mb; limit: 2048Mb)
SUMMARY: libFuzzer: out-of-memory
Error: Fuzz target exited with exit status: 71

It is NOT a defect in solve_milp — evidence, not assertion

This was investigated with a two-arm experiment rather than by reading code alone.

1. The live heap at OOM is 27.7 MB. From the job's own ASan report:

Live Heap Allocations: 27727651 bytes in 8340 chunks;
  quarantined: 155736296 bytes in 2104213 chunks;
  2232955 other chunks; total chunks: 4345508

8,340 live chunks after 185,235 executions. Any per-call leak would put that in the
hundreds of thousands. 4.3M chunks are retained by the sanitizer allocator while only
27.7 MB is live.

2. RSS tracks execution count, not input size. RSS was already 207 MB while
libFuzzer's max input length was 21 bytes:

#4281    lim: 21    rss: 207Mb
#98083   lim: 580   rss: 1035Mb
#185235  lim: 1240  rss: 2020Mb

Throughput collapsed 7772 → 311 exec/s — the signature of a growing allocator working set.

3. Same workload, ASan on vs off. A standalone driver building the same MILP shape:

arm	iterations	RSS	live heap
no ASan	20,000	12.5 MB, flat after warm-up	—
ASan	120,000	582 MB	172 KB, constant from iter ~2,000 on

4. Turning down sanitizer bookkeeping alone cuts RSS 3×, with zero code change:

ASAN_OPTIONS	RSS @ 20k iters
defaults	417 MB
quarantine_size_mb=16	187 MB
quarantine_size_mb=16:malloc_context_size=2	140 MB

solve_milp is bounded by counts, never by values — independently re-checked against
the source:

• milp.rs:67 — .filter(|(_, t)| t.period_ps > 0), so the zero-period
  divide-by-zero hypothesis is ruled out, not merely untested.
• milp.rs:95 — (0..num_valid * num_procs) ≤ 8 × 4 = 32 binaries.
• milp.rs:161 — vec![0.0_f64; num_procs], ≤ 4.
• memory_bytes (the only u32-wide fuzz field) is never read by solve_milp — its
  sole occurrence in the file is milp.rs:247, inside the test module.

The saved artifact oom-c5aafda340317c72f5ba9eb2431f0c189d159c9d (76 bytes) is not a
reproducer — it is whichever input happened to be executing when the RSS watchdog fired.
Replaying it will not OOM. Worth stating explicitly so nobody burns a day on it.

Why only this target: it is the only one running a C++ HiGHS MIP presolve + branch-and-bound
per iteration, i.e. thousands of small new/delete per exec, each becoming a quarantined
ASan chunk.

Fix

.github/workflows/fuzz-nightly.yml:62 currently passes only -max_total_time=3600 -timeout=10
and sets no ASAN_OPTIONS.

1. -max_len=128. The harness consumes at most 8 tasks (~7 bytes each) + 4 processors
   (~5 bytes) ≈ 80 bytes of entropy; letting libFuzzer grow to 1240-byte inputs is wasted
   work that also inflates per-exec churn. This improves fuzzing efficiency as well as memory.
2. ASAN_OPTIONS: quarantine_size_mb=32:malloc_context_size=5 — measured 3× reduction.
   Deep allocation stacks buy little for a target whose only invariant is "must not panic."
3. -rss_limit_mb=4096 as headroom. Blunt; growth decelerates but does not fully plateau,
   so this buys room rather than a guarantee. Prefer 1 and 2.

The part that outlives this bug

A nightly that has never once passed should not be able to stay quiet for two months. Worth
deciding: either make these workflows report failure somewhere a human sees (issue-on-fail,
notification), or accept they are advisory and stop presenting them as assurance. Related:
#358, where a gate reported passed: 20 / failed: 0 over a scope that excluded the very
artifact it was meant to check — same shape, different surface.

Verified against spar at 457592f; workflow line and history re-checked directly.

[avrabe]

Correcting two numbers in my own title, both understatements, found while
building the fix in #363.

It is worse than filed, and older. I wrote "60/60 since 2026-06-01". Querying
the full run history:

$ gh run list --workflow=fuzz-nightly.yml --limit 300 --json conclusion,createdAt
runs=97  oldest=2026-04-25T05:07:29Z  newest=2026-07-30T05:34:17Z
conclusions=failure:97

97 of 97, back to 2026-04-25. There is no passing run anywhere in the
window the API still returns — the true start date may be earlier still. My "60"
was the count for the other red nightly (#362, trace-fixtures), which genuinely
is 60/60 since 2026-06-01. I conflated the two.

But it is also narrower than filed. The workflow-level conclusion hid the
per-leg breakdown. Only fuzz_scheduler_solver fails:

leg	last 97 runs
fuzz_aadl_parse	success
fuzz_scheduler_solver	failure
fuzz_codegen_roundtrip	success

That changes the fix. The obvious patch — appending -max_len=128 to the shared
run line — would have capped two healthy fuzzers that parse arbitrarily long
AADL/WIT source text, silently shrinking their reachable input space to fix a
third leg's bug. #363 puts the flag in matrix.include instead.

The diagnosis, confirmed from ASan's own accounting

used: 2056Mb; limit: 2048Mb
Live Heap:    27.7 MB in     8,340 chunks
quarantined: 155.7 MB in 2,104,213 chunks
total chunks:              4,345,508

Live plus quarantined contents are 183 MB of 2056 MB. The other ~1.87 GB is
ASan's per-chunk bookkeeping over 4.3M chunks — so it is a product of two terms,
and #363 bounds both (quarantine_size_mb the count, malloc_context_size the
per-chunk cost).

oom-c5aafda340317c72f5ba9eb2431f0c189d159c9d is not a reproducer

The saved artifact is 267 bytes and is merely whichever input was executing when
the process crossed the RSS limit. Replaying it proves nothing. Flagging it so
nobody loses a day treating it as the trigger.

🤖 Generated with Claude Code

[avrabe]

Oracle satisfied — both sides

Nightly-equivalent dispatch on fix/fuzz-nightly-oom: run 30560598236 — conclusion: success.

The two-sided oracle from #363, restated verbatim there as "1. fuzz_scheduler_solver must flip failure → success, and 2. fuzz_aadl_parse and fuzz_codegen_roundtrip must stay success. A green run where a parser leg regressed is not a fix."

target	before	this run
fuzz_scheduler_solver	failure (OOM, 97/97 red since 2026-04-25)	success
fuzz_aadl_parse	success	success
fuzz_codegen_roundtrip	success	success

All three ran the full 1h budget rather than exiting early, so the green is not a fail-fast artifact.

Grounding note: this is the dispatch result, not yet the scheduled nightly. The nightly on main stays the real gate and won't have run until #363 lands.