Actually fuzz on a schedule, instead of only replaying the corpus - #96
Merged
Conversation
The 1651 inputs in data/fuzz/api are replayed by the test suite on every run, which is a regression guard and by construction can only find what has already been found. Nothing was doing the finding. fuzz_api is even built in CI already, by the sanitizers job, because it is gated on the compiler being clang -- it was just never run as a fuzzer. It is worth running: a 30 second local session produced 26 inputs, 20 of which survived -merge=1 as coverage-increasing. The corpus is nowhere near saturated, and the defects this container has had -- #63, #65 through #70, #74 -- were exception safety and aliasing bugs of exactly the kind this reaches. The job seeds from data/fuzz/api but writes new inputs to a scratch directory, since libFuzzer writes to the first corpus directory it is given, so a run can never modify what is committed. Afterwards it does the same -merge=1 that scripts/fuzz_merge.sh does locally, so the uploaded artifact is the handful of inputs that add coverage rather than everything the fuzzer happened to keep. A crash fails the job and uploads the reproducer. Committing what it finds stays a human decision, documented in CONTRIBUTING.md along with the local workflow: a job that pushes to the repository needs write access it has no other reason to have, and the corpus is small on purpose. Also drops '-isystem /usr/lib64/clang/14.0.0/include/' from the fuzz arguments. That is a Fedora path for a clang that is eight major versions old, it exists on no CI runner, and it was only harmless because clang ignores an include directory that is not there. See #86. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
martinus
force-pushed
the
scheduled-fuzzing
branch
from
August 5, 2026 03:40
e0538c9 to
c57bfef
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #86.
Why
The 1651 inputs in
data/fuzz/apiare replayed by the test suite on every run. That is a regression guard and by construction can only find what has already been found — nothing was doing the finding.fuzz_apiis even built in CI already, by thesanitizersjob, because it is gated on the compiler being clang. It was just never run as a fuzzer.It is worth running. A 30 second local session produced 26 inputs, 20 of which survived
-merge=1as coverage-increasing. The corpus is nowhere near saturated, and the defects this container has had — #63, #65 through #70, #74 — were exception-safety and aliasing bugs of exactly the kind a fuzzer reaches.How
A nightly job (also startable by hand from the Actions tab, with a duration) builds
fuzz_apiunder clang and runs it for ten minutes.It cannot touch the committed corpus. libFuzzer writes new inputs to the first corpus directory it is given, so the job passes a scratch directory first and
data/fuzz/apisecond — seeded from it, never writing to it. I verified this locally: after a 30 second run,git statusondata/fuzz/apiwas clean and the file count was unchanged at 1651.What it uploads is minimized. Afterwards it runs the same
-merge=1thatscripts/fuzz_merge.shdoes locally, then diffs against the committed corpus, so thecorpus-newartifact is the handful of inputs that actually add coverage rather than everything the fuzzer happened to keep. Without that the artifact would grow without bound.A crash fails the job loudly and uploads the reproducer as
fuzz-crash.The corpus policy, which the issue asked to decide
New entries are committed by hand, never by CI. The job uploads and stops. A job that pushes to the repository — or opens a PR — needs write access it has no other reason to have, and this corpus is small on purpose. Documented in
CONTRIBUTING.md, which had no fuzzing section at all despitescripts/fuzz_run.shandscripts/fuzz_merge.shexisting, along with how to run both locally and what to do with a crash.Also
Drops
-isystem /usr/lib64/clang/14.0.0/include/from the fuzz compile arguments. That is a Fedora path for a clang eight major versions old; it exists on no CI runner and was only harmless because clang silently ignores an include directory that is not there. Thesanitizersjob buildsfuzz_api, so this PR's own CI covers that change.What this PR's CI does not cover
fuzz.ymlonly triggers onscheduleandworkflow_dispatch, so none of it runs on this PR — same asrelease.yml. That is why I gave it aworkflow_dispatchtrigger with a duration input: once this is merged I intend to dispatch a short run to prove the whole thing end to end before trusting the nightly. The parts I could verify locally are the fuzz, merge and diff pipeline, and that the target still builds without the stale include path.🤖 Generated with Claude Code