Fuzzing harnesses link the real target library (fixes #151) - #152
Merged
Conversation
The live-fuzzing system prompt told the model to implement `int LLVMFuzzerTestOneInput(...)` with no `extern "C"`. The harness is compiled as C++, so the symbol was name-mangled and every link failed with `undefined reference to LLVMFuzzerTestOneInput` — the agent burned its whole budget on compile-repair and never fuzzed (fuzz_seconds=0) on libpng. zlib happened to compile, masking the issue. Spell out the C-linkage requirement (and extern "C" guarding of C headers) so harnesses link against libFuzzer's runtime. Caught during the haiku baseline sweep: libpng rep0 = 12/12 compile failures, 0 fuzzing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…#148) start_fuzzing re-ran each crash via sandbox.exec("/out/<target> <crash_path>"), but crash_path came from collect_crashes(), which copies crashes OUT to a host tempdir (/var/folders/.../crashes-*). sandbox.exec runs INSIDE the container, so the binary couldn't find the host path and failed with "directory does not exist" -- and that error string was then stored as the crash's ASAN output and hashed as a unique crash. Result: every crash was unretrievable/untriageable and the per-host-path error inflated eff_vulns with noise (0-coverage "artifacts"). Copy each crash back into the container and re-run the in-container path, so get_crash_info returns the real ASAN and dedup works on real stacks. Adds a deterministic regression test (mocked sandbox + campaign). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
) final_line_coverage_pct was hardcoded 0.0 and never populated, so every fuzzing sample reported 0% coverage -- we couldn't tell "harness reaches target code" from "harness crashes on its own input". (True line-% needs a coverage- instrumented build, a bigger change.) Parse libFuzzer's running "cov: N" counter (covered PCs/edges) as a tractable proxy: FuzzingStats.coverage_pcs = max cov: seen. Thread it through start_fuzzing (peak across cycles, per-cycle Logfire attr), the session result, and the score metadata so coverage_pcs is queryable. coverage_pcs>0 means the harness actually exercised target code; 0 flags the artifact self-crashers. Unit tests for the parse and the threading. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The harness compile command only set `-L` search paths; it never put the target's headers on the include path or linked the static library `.a`. As a result agents could not call the real library: real calls hit "undefined reference", so the surviving harnesses either (a) stub the library functions themselves or (b) wrap the includes in `#if __has_include(...)` guards that — with the wrong include path — silently compile the harness body to a no-op. Either way the fuzzer ran against the agent's own code, not the target, which structurally explains the persistent near-zero real crashes. Fix, derived from each target's build.sh link step: - Add HARNESS_LINK_RECIPES (per-target include dirs + static .a candidates + dependency flags) and resolve_harness_link(), which discovers the first existing .a at session start (mirrors build.sh's own fallback logic). - compile_harness() now uses the resolved include + link flags, so every harness links the real instrumented library. - system.prompt: instruct the agent that the library is pre-linked — include the real headers unconditionally, call real functions by their real names (no invented OSS_FUZZ_* prefixes), and never stub or `__has_include`-guard the library. Validated in-container on all four Category-1 targets: the recovered harnesses now link the real library (nm shows T xmlReadMemory / uncompress / jpeg_read_header; libpng real png_* link clean) and libxml2 executes real parser coverage, where the old command produced a no-op. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The experiment runner processes its grid sequentially (one inspect_eval at a time), so a single invocation uses ~one core. This script drives concurrency (up to $CAP shards at once) and INTERLEAVES models+targets so every (model,target) combo advances together — balanced data even on partial rounds. Self-heals from the failure modes we hit: Docker down (orbctl start), disk full (emergency prune + background janitor), external volume unmount (optional remount), transient sample failures (--retry-failed up to $K). Resumable via per-shard manifests; loops until a stop file appears. Parameterized via env vars and path-portable (derives the scaffold dir from its own location). Also gitignore the auto-generated per-shard sweep TOMLs (keeps the 4 committed configs). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two infra bugs found while babysitting the linked run, both losing/skewing data: 1. CrashSummary had cwe/severity/crash_file typed `str | None` but WITHOUT defaults -> still required in pydantic v2. save_session_memory() builds it without them, so every crash-finding sample raised ValidationError; the call site was unguarded, so the whole sample FAILED (status=error) and its data (tokens, coverage, the crash) was lost. Give the fields `= None` defaults and wrap save_session_memory in try/except so a memory-write can never fail a sample. 2. fuzzing_sweep_loop.sh launched group->target->model, so the first CAP shards were all libpng+libxml2; combined with the janitor job silently eating one CAP slot, libjpeg-turbo and zlib were starved (zero shards in 3h). Reorder to group->model->target (target innermost spans all targets in the first wave) and exclude the janitor PID from the concurrency gate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
macOS writes ._memory.json beside memory.json on the NTFS SSD when the memory feature persists per-target state; the file lands in the docker build context and breaks orbstack's context transfer (failed to xattr ... operation not permitted -> build rc=1), which silently failed 8/10 libjpeg-turbo reps. Sweep ._* from the targets tree before every build attempt.
Issue #114 resolved to "Logfire is the primary research dashboard" — no bespoke Next.js app, no Streamlit `viz/` rebuild. The one unchecked load-bearing decision was "finish the structured-attribute follow-up so domain metrics land as Logfire attributes (prerequisite for Logfire owning the curves)." Reading down the live Logfire project surfaced the exact gap: the run-level `eval sample` spans and per-cycle `fuzz cycle complete` records were rich (efficiency metrics, coverage, crashes) but carried **no `model` attribute anywhere** — so the core research crux ("vulns per walltime as a function of model and architecture") could not be charted, and concurrent runs of the same target collapsed into one tangled line. Changes: - telemetry.py: add `model` to `_sample_attributes` (eval-sample spans), derived from the busiest `model_usage` key so grader/critic calls don't mislabel the run. - tasks/fuzzing.py: stash run identity (`model`, `sample_id`, `epoch`) into session_state at solver start and emit it on every `fuzz cycle complete` record, so live curves break down by model and separate per run. - tasks/fuzzing.py: fix `log.warning` -> `log.warn` (the facade exposes `warn`; the call would have raised AttributeError in its own error path). - docs/logfire-dashboard.md: saved Logfire SQL queries for the core views (vulns-per-walltime leaderboard by model x target, live exec/s curves, cumulative crashes, coverage growth, failure rate). All validated against the live project. Documents the telemetry contract and a pre-existing test-telemetry-pollution issue. Built on top of PR #152 (real-library linking), which is where the first valid fuzzing data — and the `coverage_pcs` signal — actually comes from. 954 tests pass; ruff/ty clean on changed files. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Member
|
Heads-up: I pushed a commit onto this branch ( What it adds
One change inside your code: Reading down Logfire is what surfaced the gap: run-level spans were already rich (the #135 efficiency block lands fine), but nothing carried Pre-existing issue I noticed (not fixing here): the pytest suite emits 954 tests pass; ruff/ty clean on changed files. |
Replace the per-prefix sweep-config ignore rules with an allowlist: ignore all scaffold/experiments/*.toml except the four hand-written configs. Silences the generated per-shard/mini/validate run TOMLs (opus-*, sonnet-*, fuzzing-mini-*, validate-*, ...) that were cluttering git status.
…emory.json Allowlist the committed canonical run summaries (fuzzing-baseline[-v2], fuzzing-test, patching-model-sweep[-v2]) and ignore every other results/ dir (throwaway sweeps; full .eval logs already ignored + live in DO Spaces). Also ignore generated targets/*/memory.json run state. git status is now clean.
Logfire's custom-dashboard creator imports a JSON definition (Dashboards → Custom → Import JSON), not raw SQL — the SQL lives embedded inside each panel. Add `docs/logfire-dashboard.json` so the whole dashboard imports in one step instead of pasting five queries by hand. Schema reverse-engineered from a "Download dashboard as code" export (undocumented + version-specific). Seven panels in two sections: - Research crux (run-level, from `eval sample` spans): total unique crashes (Values), vulns-per-fuzz-hour by target and by model (BarChart), run failure rate by target. - Live curves (per cycle, from `fuzz cycle complete`): execs/sec, peak coverage PCs, and unique crashes over time, grouped by target (TimeSeriesChart with time_bucket($resolution, ...)). All seven panel queries validated against the live project. The by-model panel uses the #114 `model` attribute and shows "(model not recorded)" for pre-#114 historical runs until new runs land. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Member
|
Follow-up to the #114 work above: added To use it: Logfire → Dashboards → Custom → Import JSON → upload
The custom-dashboard creator imports JSON (SQL is embedded per-panel), not raw SQL — schema reverse-engineered from a "Download dashboard as code" export. The by-model panel shows "(model not recorded)" for pre-#114 runs and splits by model once new runs land. |
Member
|
@copilot resolve the merge conflicts in this pull request |
Copilot stopped work on behalf of
quinn-dougherty due to an error
June 22, 2026 18:51
quinn-dougherty
approved these changes
Jun 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #151.
Live-fuzzing harnesses never linked or exercised the real target library — the harness compile command set
-Lsearch paths but never linked the built.a, and used the wrong include path. Agents either stubbed the library or__has_include-guarded it into a no-op (both compiled rc=0, so the failure was silent). Every prior fuzzing result was measuring the wrong code. See #151 for the full diagnosis.What's here (commits off
mainsince #149 merged)HARNESS_LINK_RECIPES+resolve_harness_link()(per-target include dirs + static.a+ dep flags, from eachbuild.sh);compile_harnessuses them.OSS_FUZZ_prefix), never stub or__has_include-guard.cov:PCs (needed to even see this bug).extern "C"prompt requirement.scripts/fuzzing_sweep_loop.sh— resilient multi-core sweep orchestration (runner is otherwise sequential); gitignore auto-generated shard configs.Validation
In-container on all 4 Cat-1 targets the harness now links the real library (
nm→T xmlReadMemory/uncompress/jpeg_read_header; libpng realpng_*link clean); live run shows real libxml2coverage_pcs21–54 where the old command was a no-op. 954 tests pass; ruff/ty clean.Note
Bundles supporting fuzzing-blocker fixes (coverage, crash-retrieval, extern C) that landed on the old
issue-148branch after PR #149 was merged and so never reachedmain. Follow-up: agents call real functions with wrong args (no signatures in context) → #150.🤖 Generated with Claude Code