docs(coin-gym): reference the reproducible baseline-vs-team measurement - #3152
Merged
rysweet merged 1 commit intoJul 8, 2026
Conversation
Records the LOCAL COIN Gym harness's reproducible single-model baseline vs. multi-agent team result as a durable reference doc under docs/research/. On the bundled sample target set both arms reach 60% (3/5), while the team's abstention gate lifts precision from 60% to 100% (+40 pts) at equal reach — the central precision/over-claiming trade-off the harness is built to expose. The numbers are a deterministic, unit-tested property (pinned by coin_gym::tests_cli::execute_run_baseline_vs_team_shows_precision_tradeoff), so this is a maintained reference — not a point-in-time report — that a future PR changing the strategies or fixture is expected to update (G4-compliant). - add docs/research/coin-gym-baseline-vs-team-measurement.md (methodology, per-target breakdown, reproduction commands, leaderboard caveat, what remains for Phase 3) - register it in mkdocs.yml nav - cross-link from the how-to and the skwaq gym study Docs-only; LOCAL-ONLY (nothing submitted externally). A real leaderboard grade still depends on the Phase-3 azlin VM (#2823). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
4 tasks
rysweet
deleted the
engineer/build-a-local-coin-benchmark-harness-and-a-self-09e65e35-1783510831-cd46fd
branch
July 8, 2026 12:58
This was referenced Jul 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
The COIN Gym harness (Phases 1/2/4/5) is merged to
main, but the goal'sdone-criterion — an empirical baseline-vs-team measurement — was not yet
recorded. This PR runs the merged harness end-to-end and records its
reproducible single-model-baseline vs. multi-agent-team result as a
durable reference doc.
Result (bundled sample target set, offline scaffold / mock oracle):
baseline(single model)R:3/W:2/A:0/T:0/N:0/E:0team(multi-agent)R:3/W:0/A:2/T:0/N:0/E:0The team measurably beats the baseline on precision (+40 pts) at equal reach.
The abstention gate abstains on the two low-confidence targets (
liboqs-kem-88conf 0.44,
zstd-huf-1207conf 0.35) where the baseline submits wrong inputs.Reproduced end-to-end at commit
ad4c8032(main tip, 2026-07-08).Durable-documentation (G4) note
The only added doc —
docs/research/coin-gym-baseline-vs-team-measurement.md—is a unit-test-pinned durable reference (numbers asserted by
coin_gym::tests_cli::execute_run_baseline_vs_team_shows_precision_tradeoff),not a point-in-time report: a future strategy/fixture change is expected to update
it. Its title surface carries none of the report-kind markers checked by
scan_no_point_in_time_report_docs, and it lives underdocs/research/(not areserved report dir). The point-in-time milestone finding (numbers + commit SHA)
is offloaded to tracking issue #2823:
#2823 (comment)
(
docs/research/coin-benchmark-phase1.mdis a pre-existing file onmain, notin this diff.)
Merge readiness
QA-team evidence
gadugi-testscenario to exercise (internal-only justification below).mkdocs build --strict(doc build) andcargo test -p simard --lib coin_gym::tests_cli::execute_run_baseline_vs_team_shows_precision_tradeoff(pins the exact reference numbers the doc records).
mkdocs build --strictexit 0 (nowarnings/errors);
test result: ok. 1 passed.coin-gym run "Claude Opus 4.6" --strategy baselineand... --strategy teamon the bundledsample_snapshot.json.(
R:3/W:2), team 60.0%/100.0% (R:3/W:0/A:2); numbers reproduced in the doc.mkdocs.ymlnav only; there is no code/CLI/config/API behavior change, so theauthoritative validators are the strict doc build and the existing
number-pinning unit test, both green.
Documentation
docs/research/coin-gym-baseline-vs-team-measurement.md;edited
docs/howto/run-the-coin-gym-harness.md,docs/research/coin-benchmark-and-skwaq-study.md, andmkdocs.yml(nav).cross-linked both ways (how-to ⇄ measurement, study ⇄ measurement).
Quality-audit
from source of truth? VALIDATE: re-derived every number from a fresh
coin-gym run(baseline 60/60, team 60/100; family splits frontier50/50→50/100, non-trivial 66.7/66.7→66.7/100; histograms
R:3/W:2andR:3/W:0/A:2), the fixture confidences (0.82/0.71/0.66/0.44/0.35fromsample_snapshot.json), and the leaderboard figures (30.0/52.5fromleaderboard.rs). FIX: none — all matched exactly.orphan page, or missing reciprocal link? VALIDATE: resolved all 3 relative
.mdlinks to existing files, confirmed reciprocal links in how-to/study/nav,ran
mkdocs build --strict(exit 0, no warnings/errors). FIX: none.overclaim a live capability result or trip the report-doc scan? VALIDATE: title
surface has zero report-kind markers; explicit
offline scaffold/LOCAL-ONLYcaveats, an "illustrative only" leaderboard note, and a "what this does and does
not establish" section scoping the result to offline control-flow. FIX: none.
correctness/security findings.
each ended clean with no findings, so the audit is complete. Landed as commit
f4a4e935(docs-only).CI
gh pr checks 3152.build,pre-commit,install-real,e2e-dashboard,cargo-audit,cargo-deny,cargo-vet,npm-audit, GitGuardian; 0failures, 0 pending. Runs:
https://github.com/rysweet/Simard/actions/runs/28941803288 ·
https://github.com/rysweet/Simard/actions/runs/28941803054 ·
https://github.com/rysweet/Simard/actions/runs/28941773879
(
rust-only-gate,cargo fmt --check,clippy --release) and pre-push(
clippy --all-targets --all-features --locked, race test subset) also Passed.Scope
gh pr view 3152 --json files→ exactly 4 files(1 added doc +
mkdocs.yml+ 2 doc edits).git diff --stat: 3 files changed,10 insertions(+), 1 deletion(-), plus the new doc.
reference doc or a link/nav entry pointing to it.
Verdict
Explicit call: this PR is READY TO MERGE (it is not a draft and is not
blocked). Rationale: all six criteria above are satisfied with concrete evidence
— QA validated via
mkdocs build --strict+ a passing number-pinning unit test(docs-only, internal-only justification given), the new reference doc is
registered and cross-linked, three independent quality-audit cycles ended clean
(zero critical/high/medium), CI is 100% green with run links, and the diff is
focused with no unrelated edits. Advisory G4 is addressed (durable
unit-test-pinned reference; point-in-time milestone recorded on issue #2823);
live leaderboard grading is separately tracked as Phase 3 (#2823) and does not
gate this docs-only PR.
LOCAL-ONLY: nothing is submitted externally or entered on any leaderboard.