Skip to content

docs(coin-gym): reference the reproducible baseline-vs-team measurement - #3152

Merged
rysweet merged 1 commit into
mainfrom
engineer/build-a-local-coin-benchmark-harness-and-a-self-09e65e35-1783510831-cd46fd
Jul 8, 2026
Merged

docs(coin-gym): reference the reproducible baseline-vs-team measurement#3152
rysweet merged 1 commit into
mainfrom
engineer/build-a-local-coin-benchmark-harness-and-a-self-09e65e35-1783510831-cd46fd

Conversation

@rysweet

@rysweet rysweet commented Jul 8, 2026

Copy link
Copy Markdown
Owner

What & why

The COIN Gym harness (Phases 1/2/4/5) is merged to main, but the goal's
done-criterion — an empirical baseline-vs-team measurement — was not yet
recorded. This PR runs the merged harness end-to-end and records its
reproducible single-model-baseline vs. multi-agent-team result as a
durable reference doc.

Result (bundled sample target set, offline scaffold / mock oracle):

Arm Reach Precision Histogram
baseline (single model) 60.0% (3/5) 60.0% (3/5) R:3/W:2/A:0/T:0/N:0/E:0
team (multi-agent) 60.0% (3/5) 100.0% (3/3) R:3/W:0/A:2/T:0/N:0/E:0

The team measurably beats the baseline on precision (+40 pts) at equal reach.
The abstention gate abstains on the two low-confidence targets (liboqs-kem-88
conf 0.44, zstd-huf-1207 conf 0.35) where the baseline submits wrong inputs.
Reproduced end-to-end at commit ad4c8032 (main tip, 2026-07-08).

Durable-documentation (G4) note

The only added doc — docs/research/coin-gym-baseline-vs-team-measurement.md
is a unit-test-pinned durable reference (numbers asserted by
coin_gym::tests_cli::execute_run_baseline_vs_team_shows_precision_tradeoff),
not a point-in-time report: a future strategy/fixture change is expected to update
it. Its title surface carries none of the report-kind markers checked by
scan_no_point_in_time_report_docs, and it lives under docs/research/ (not a
reserved report dir). The point-in-time milestone finding (numbers + commit SHA)
is offloaded to tracking issue #2823:
#2823 (comment)
(docs/research/coin-benchmark-phase1.md is a pre-existing file on main, not
in this diff.)

Merge readiness

QA-team evidence

  • Scenario files: none — docs-only change; no new runtime/user surface for a
    gadugi-test scenario to exercise (internal-only justification below).
  • Validation command: mkdocs build --strict (doc build) and
    cargo test -p simard --lib coin_gym::tests_cli::execute_run_baseline_vs_team_shows_precision_tradeoff
    (pins the exact reference numbers the doc records).
  • Validation result: passedmkdocs build --strict exit 0 (no
    warnings/errors); test result: ok. 1 passed.
  • Run command: coin-gym run "Claude Opus 4.6" --strategy baseline and
    ... --strategy team on the bundled sample_snapshot.json.
  • Run target: local (offline scaffold, mock oracle).
  • Run result: passed — baseline reach/precision 60.0%/60.0%
    (R:3/W:2), team 60.0%/100.0% (R:3/W:0/A:2); numbers reproduced in the doc.
  • Evidence location: PR result table above + issue COIN Gym Phase 3 — provision azlin VM (DefenderATEVET17) + smoke coin evaluate #2823 comment (linked).
  • Rationale for no gadugi scenario: the changed surfaces are three docs +
    mkdocs.yml nav only; there is no code/CLI/config/API behavior change, so the
    authoritative validators are the strict doc build and the existing
    number-pinning unit test, both green.

Documentation

  • User-facing docs impact: yes (this PR is the doc).
  • Updated docs: added docs/research/coin-gym-baseline-vs-team-measurement.md;
    edited docs/howto/run-the-coin-gym-harness.md,
    docs/research/coin-benchmark-and-skwaq-study.md, and mkdocs.yml (nav).
  • PR description links added: new doc registered in the Research nav and
    cross-linked both ways (how-to ⇄ measurement, study ⇄ measurement).
  • Rationale if not applicable: n/a.

Quality-audit

  • Cycle 1 summary — numerical/claim accuracy. SEEK: could any figure drift
    from source of truth? VALIDATE: re-derived every number from a fresh
    coin-gym run (baseline 60/60, team 60/100; family splits frontier
    50/50→50/100, non-trivial 66.7/66.7→66.7/100; histograms R:3/W:2 and
    R:3/W:0/A:2), the fixture confidences (0.82/0.71/0.66/0.44/0.35 from
    sample_snapshot.json), and the leaderboard figures (30.0/52.5 from
    leaderboard.rs). FIX: none — all matched exactly.
  • Cycle 2 summary — link/anchor/build integrity. SEEK: any broken link,
    orphan page, or missing reciprocal link? VALIDATE: resolved all 3 relative
    .md links to existing files, confirmed reciprocal links in how-to/study/nav,
    ran mkdocs build --strict (exit 0, no warnings/errors). FIX: none.
  • Cycle 3 summary — G4 durability + no-overclaiming. SEEK: does the doc
    overclaim a live capability result or trip the report-doc scan? VALIDATE: title
    surface has zero report-kind markers; explicit offline scaffold / LOCAL-ONLY
    caveats, an "illustrative only" leaderboard note, and a "what this does and does
    not establish" section scoping the result to offline control-flow. FIX: none.
  • Additional cycles: none needed.
  • Final clean cycle: cycle 3 — zero critical/high; zero medium
    correctness/security findings.
  • Fixes followed default-workflow: n/a (no fixes were required; all cycles clean).
  • Convergence summary: three independent audits (accuracy, links/build, policy)
    each ended clean with no findings, so the audit is complete. Landed as commit
    f4a4e935 (docs-only).

CI

Scope

  • Changed files reviewed: gh pr view 3152 --json files → exactly 4 files
    (1 added doc + mkdocs.yml + 2 doc edits). git diff --stat: 3 files changed,
    10 insertions(+), 1 deletion(-), plus the new doc.
  • Unrelated changes: none — docs-only; no code changes; every edit is the new
    reference doc or a link/nav entry pointing to it.

Verdict

  • Merge-ready: yes
  • Remaining blockers: none

Explicit call: this PR is READY TO MERGE (it is not a draft and is not
blocked). Rationale: all six criteria above are satisfied with concrete evidence
— QA validated via mkdocs build --strict + a passing number-pinning unit test
(docs-only, internal-only justification given), the new reference doc is
registered and cross-linked, three independent quality-audit cycles ended clean
(zero critical/high/medium), CI is 100% green with run links, and the diff is
focused with no unrelated edits. Advisory G4 is addressed (durable
unit-test-pinned reference; point-in-time milestone recorded on issue #2823);
live leaderboard grading is separately tracked as Phase 3 (#2823) and does not
gate this docs-only PR.

LOCAL-ONLY: nothing is submitted externally or entered on any leaderboard.

Records the LOCAL COIN Gym harness's reproducible single-model baseline vs.
multi-agent team result as a durable reference doc under docs/research/. On the
bundled sample target set both arms reach 60% (3/5), while the team's abstention
gate lifts precision from 60% to 100% (+40 pts) at equal reach — the central
precision/over-claiming trade-off the harness is built to expose.

The numbers are a deterministic, unit-tested property (pinned by
coin_gym::tests_cli::execute_run_baseline_vs_team_shows_precision_tradeoff), so
this is a maintained reference — not a point-in-time report — that a future PR
changing the strategies or fixture is expected to update (G4-compliant).

- add docs/research/coin-gym-baseline-vs-team-measurement.md (methodology,
  per-target breakdown, reproduction commands, leaderboard caveat, what remains
  for Phase 3)
- register it in mkdocs.yml nav
- cross-link from the how-to and the skwaq gym study

Docs-only; LOCAL-ONLY (nothing submitted externally). A real leaderboard grade
still depends on the Phase-3 azlin VM (#2823).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@rysweet
rysweet merged commit 08a3d36 into main Jul 8, 2026
16 checks passed
@rysweet
rysweet deleted the engineer/build-a-local-coin-benchmark-harness-and-a-self-09e65e35-1783510831-cd46fd branch July 8, 2026 12:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant