Skip to content

CI: run the fuzzer on a schedule instead of only replaying the corpus #86

Description

@martinus

What

Run the libFuzzer target on a schedule and feed what it finds back into the committed corpus. Today the fuzzer is only ever replayed, never run.

Why

The corpus is replayed, but nothing explores. test/unit/fuzz_corpus.cpp replays the 1651 committed inputs under data/fuzz/api as a normal test, which is a good regression guard but by construction can only find what has already been found. The actual libFuzzer binary (fuzz_api, gated on compiler.get_id() == 'clang' in test/meson.build) is already built by the sanitizers CI job, since that job uses clang. It is simply never run as a fuzzer.

The corpus is nowhere near saturated. A 30 second local run added 11 new corpus units:

meson setup /tmp/bdfuzz            # with CXX=clang++
ninja -C /tmp/bdfuzz test/fuzz_api # note the target path, plain 'fuzz_api' is not a ninja target
/tmp/bdfuzz/test/fuzz_api -max_total_time=30 -print_final_stats=1 data/fuzz/api
# => 5399 execs, cov: 1213, ft: 8818, new_units_added: 11

Eleven new inputs in half a minute means a nightly run of ten minutes has real headroom to find more. Given the container's history — #63, #65 through #70 and #74 were all exception-safety and aliasing defects found by looking rather than by users — this is the highest-yield automated check available.

One thing must be fixed first. test/meson.build hardcodes -isystem /usr/lib64/clang/14.0.0/include/ in fuzz_cpp_args. That is a Fedora-specific path for clang 14; the local toolchain is clang 22 and GitHub runners have neither. It is currently harmless because clang ignores a non-existent -isystem directory, but it is stale and should go.

Work

  • Remove the hardcoded -isystem /usr/lib64/clang/14.0.0/include/ from fuzz_cpp_args in test/meson.build, and confirm fuzz_api still builds under clang locally and on a runner.
  • Add a scheduled workflow (nightly, or weekly if that is preferred) that builds fuzz_api with clang and runs it for a bounded time — start with -max_total_time=600 — seeded from data/fuzz/api.
  • On a crash or timeout, upload the reproducer as an artifact and fail the job loudly. A crash here is a real bug and should not be silent.
  • Decide and document the corpus policy. Options: have the job open a PR with newly discovered units, or minimise with -merge=1 and commit periodically by hand. Do not let the job push to main directly. Whatever is chosen, note it in CONTRIBUTING.md alongside the existing scripts/fuzz_run.sh workflow so it is discoverable.
  • Consider having the job run -merge=1 into a temporary directory first so the committed corpus stays minimal rather than growing without bound.

Acceptance criteria

  • A scheduled run builds and fuzzes without the stale include path.
  • An artificially introduced bug (for example, temporarily breaking a bounds check) is caught by the job and produces a downloadable reproducer.
  • The committed corpus does not grow unboundedly on every run.
  • scripts/fuzz_run.sh still works for local runs.

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions