Skip to content

test: YAML + CLI fuzzer targets + empirical evidence for 5 silent-accept bugs#160

Merged
avrabe merged 1 commit intomainfrom
test/yaml-cli-fuzzers
Apr 22, 2026
Merged

test: YAML + CLI fuzzer targets + empirical evidence for 5 silent-accept bugs#160
avrabe merged 1 commit intomainfrom
test/yaml-cli-fuzzers

Conversation

@avrabe
Copy link
Copy Markdown
Contributor

@avrabe avrabe commented Apr 21, 2026

What this ships

`cargo-fuzz` + `Arbitrary` targets for rivet's artifact ingest + CLI surface:

  • `fuzz/fuzz_targets/yaml_footguns.rs` — 12 adversarial YAML variants × 5 oracles (source-substring preservation, phantom-link detection, null-ish target, serde-vs-HIR divergence, multi-doc truncation)
  • `fuzz/fuzz_targets/cli_argv.rs` — structured argv fuzzer (subprocess-based, gated on `$RIVET_BIN`)
  • `fuzz/fuzz_targets/artifact_ids.rs` — byte-round-trip through Store insert→get
  • `fuzz/examples/oracle_smoke.rs` — sandbox-safe harness; runs the same oracle against 12 hand-picked Mythos-predicted inputs via plain `cargo run`
  • `.github/workflows/fuzz.yml` — main-only + nightly cron, 15 min/target, corpus cached, crashes uploaded

Empirical findings on current main

The oracle smoke harness ran and found 5 silent-accept bugs, matching every Mythos-predicted failure mode:

Input Symptom Source-of-truth file
`target: null` phantom `Link{target:"null"}` in HIR, rivet-validate emits "broken-link" pointing at "null" yaml_hir.rs:530
`target: ~` phantom `Link{target:"~"}` yaml_hir.rs:530
`target: ""` phantom empty-target link yaml_hir.rs:530
`---` separator 2 declared artifacts become 1 in HIR, 0 diagnostics yaml_cst.rs:517
`artifact:` typo HIR returns `Ok(vec![])`, 0 diagnostics formats/generic.rs:138

These settle the arxiv:2604.13108 50%-silent-corruption claim empirically on our own corpus. 5 of 12 curated footguns silently pass.

Test plan

  • `cargo check` clean for all 3 libfuzzer targets
  • `cargo run --example oracle_smoke` reproduces the 5 findings in <1s
  • Oracle code identical between libfuzzer + smoke variants (shared helpers)
  • First nightly fuzz.yml run

🤖 Generated with Claude Code

github-actions[bot]
github-actions Bot reviewed Apr 21, 2026
View reviewed changes
Copy link
Copy Markdown

@github-actions github-actions Bot left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Performance Alert ⚠️

Possible performance regression was detected for benchmark 'Rivet Criterion Benchmarks'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 1.20.

Benchmark suite Current: 8738aad Previous: 571816f Ratio
store_insert/100 80746 ns/iter (± 914) 65346 ns/iter (± 499) 1.24
store_insert/1000 854892 ns/iter (± 7696) 702590 ns/iter (± 3980) 1.22
store_insert/10000 16034908 ns/iter (± 897633) 10122271 ns/iter (± 362822) 1.58
store_lookup/100 2142 ns/iter (± 76) 1475 ns/iter (± 9) 1.45
store_lookup/1000 25382 ns/iter (± 855) 18463 ns/iter (± 37) 1.37
store_lookup/10000 349342 ns/iter (± 7974) 265125 ns/iter (± 1804) 1.32
store_by_type/100 95 ns/iter (± 0) 75 ns/iter (± 0) 1.27
store_by_type/1000 95 ns/iter (± 0) 75 ns/iter (± 0) 1.27
store_by_type/10000 95 ns/iter (± 0) 75 ns/iter (± 0) 1.27
schema_load_and_merge 999222 ns/iter (± 16246) 769017 ns/iter (± 2168) 1.30
link_graph_build/100 164599 ns/iter (± 756) 129223 ns/iter (± 877) 1.27
link_graph_build/1000 1921416 ns/iter (± 12074) 1472888 ns/iter (± 15526) 1.30
validate/100 112358 ns/iter (± 580) 83228 ns/iter (± 360) 1.35
validate/1000 948522 ns/iter (± 4774) 732151 ns/iter (± 3463) 1.30
traceability_matrix/100 4254 ns/iter (± 48) 3199 ns/iter (± 7) 1.33
traceability_matrix/1000 58903 ns/iter (± 339) 34614 ns/iter (± 355) 1.70
traceability_matrix/10000 799409 ns/iter (± 27154) 560685 ns/iter (± 6804) 1.43
diff/100 62566 ns/iter (± 369) 47444 ns/iter (± 80) 1.32
diff/1000 688686 ns/iter (± 3265) 515789 ns/iter (± 2510) 1.34
diff/10000 9149866 ns/iter (± 582888) 6011153 ns/iter (± 47701) 1.52
query/100 808 ns/iter (± 27) 568 ns/iter (± 3) 1.42
query/1000 7413 ns/iter (± 44) 5042 ns/iter (± 10) 1.47
query/10000 109151 ns/iter (± 773) 69334 ns/iter (± 172) 1.57
document_parse/10 23343 ns/iter (± 376) 16739 ns/iter (± 69) 1.39
document_parse/100 164344 ns/iter (± 561) 117089 ns/iter (± 327) 1.40
document_parse/1000 1524974 ns/iter (± 10656) 1082997 ns/iter (± 5475) 1.41

This comment was automatically generated by workflow using github-action-benchmark.

@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 21, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@avrabe avrabe force-pushed the test/yaml-cli-fuzzers branch from 1701302 to c859444 Compare April 21, 2026 18:59
@avrabe avrabe mentioned this pull request Apr 21, 2026
Merged
5 tasks
@avrabe avrabe force-pushed the test/yaml-cli-fuzzers branch 3 times, most recently from 6061095 to 06642ed Compare April 22, 2026 04:19
@avrabe @claude
test(fuzz): YAML footgun + CLI argv + artifact-id fuzzers
8738aad 
Adds three libfuzzer-based targets under fuzz/ that empirically measure
the arxiv:2604.13108 "YAML silently corrupts ~50% of structural errors"
claim against rivet's actual artifact-ingest pipeline, plus a CLI argv
target and an id-roundtrip target.

Targets:
  * yaml_footguns  — Arbitrary-driven adversarial mutations of a known
    valid seed YAML (Norway, version-coercion, leading-zero-id,
    unquoted-date, duplicate-key, tab-indent, multi-doc, null-shorthand
    link, unknown top-level key, anchor cycle, deep nesting, control
    chars in id).  Five oracles: source-substring invariant for ids /
    types / link targets, phantom-link detection, null-ish target
    detection, serde-rejected-but-hir-accepted detection, and
    multi-document truncation detection.
  * cli_argv      — structured argv for rivet-cli subprocess; oracle
    fails on signal-death or when --format json returns success with
    non-JSON stdout.  Gated on $RIVET_BIN env var so it skips silently
    if no binary is configured.
  * artifact_ids  — arbitrary bytes as id: scalar; oracle requires
    Store::insert → Store::get to round-trip byte-exact.

Also adds fuzz/examples/oracle_smoke.rs — a non-libfuzzer harness that
runs the same oracle logic against a fixed set of Mythos-predicted
footgun inputs.  Running `cargo run --release --example oracle_smoke`
(before cargo-fuzz is available in CI) produces five findings on
current main, empirically confirming:
  - null / tilde / empty-string link targets produce phantom links
    (yaml_hir.rs:530-549 bug class)
  - multi-document YAML is silently truncated by the HIR path
    (yaml_cst.rs:517 bug class)
  - renaming `artifacts:` to a sibling key causes the HIR path to
    return Ok(vec![]) with zero diagnostics (formats/generic.rs:138)

CI: .github/workflows/fuzz.yml runs each target for 15 min on push to
main and nightly at 06:17 UTC.  continue-on-error so new crashes do
not block merges; crashes upload as workflow artifacts and the evolved
corpus is cached between runs.

REQ-052 is scoped to variant-solver fuzzing; these YAML/CLI fuzzers
verify the broader parser surface (REQ-028) and CLI surface (REQ-007).

Verifies: REQ-028, REQ-007
Refs: REQ-052

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@avrabe avrabe force-pushed the test/yaml-cli-fuzzers branch from 06642ed to 8738aad Compare April 22, 2026 04:19
@avrabe avrabe merged commit 19e8cba into main Apr 22, 2026
13 checks passed
@avrabe avrabe deleted the test/yaml-cli-fuzzers branch April 22, 2026 04:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@github-actions github-actions[bot] github-actions[bot] left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@avrabe