Skip to content

fix(ooda): machine-checkable done-gate unsticks the coin-gym goal (UNCLEAR-CRITERIA) - #4322

Open
rysweet wants to merge 1 commit into
mainfrom
fix/coin-gym-machine-checkable-done-gate-09e65e35
Open

fix(ooda): machine-checkable done-gate unsticks the coin-gym goal (UNCLEAR-CRITERIA)#4322
rysweet wants to merge 1 commit into
mainfrom
fix/coin-gym-machine-checkable-done-gate-09e65e35

Conversation

@rysweet

@rysweet rysweet commented Jul 18, 2026

Copy link
Copy Markdown
Owner

Triage & course-correction

The standing goal `build-a-local-coin-benchmark-harness-and-a-self-09e65e35`
("build a local coin benchmark harness and a self-improvement loop") was durably
`Blocked` with why=UNCLEAR-CRITERIA"no tracked PR/issue the done-gate can
verify."

Root cause: not technical. The LOCAL COIN Gym harness and its Phase-5
self-improvement loop are already built and green (src/bin/coin_gym.rs,
src/coin_gym/, gadugi scenarios, docs). The goal re-stalled every cycle only
because its finish condition had no machine-checkable definition — so each
OODA Observe tick re-saw it as unfinished and emitted NO ACTION.

Course-correction chosen (option a — make the done-gate machine-checkable),
mirroring the #4210 / kgpacks-parity fix:

  • Specs/coin-gym-benchmark-harness.md — binds the goal's finish condition
    to a single verifiable command and enumerates seven measurable done-criteria
    (target-loader, baseline/team runners, scorer, leaderboard-comparator,
    self-improvement-loop, contract-wiring), each asserted by the harness's own
    coin-gym verify self-check.
  • scripts/check-coin-gym-done-gate.sh — one command the done-gate can run
    each cycle; builds coin-gym and runs coin-gym verify, exiting 0 only when
    all seven LOCAL criteria pass. --full also runs the two outside-in gadugi
    scenarios.
  • docs/operations/coin-gym-goal-signal-2026-07-18.md — the plain-language
    operator/Signal notification (linked from mkdocs nav).

Verification

$ scripts/check-coin-gym-done-gate.sh
  [PASS] target-loader            5 pinned + 2 held-out-fresh target(s); both families present
  [PASS] baseline-runner          5 outcome(s) for 5 pinned target(s)
  [PASS] team-runner              5 outcome(s) for 5 pinned target(s)
  [PASS] scorer                   reach 60.0% / precision 60.0%; 2 family split; histogram covers 5/5
  [PASS] leaderboard-comparator   compared vs published 'GPT-5.4' (reach Δ +37.1 pts)
  [PASS] self-improvement-loop    held-out reach 0.0% → 100.0% (kept 2, rolled back 0)
  [PASS] contract-wiring          evaluate (8 args) + verify (4 args) argv present; LOCAL-ONLY=true
result: 7/7 criteria passed
✅ DONE — exit 0

Live VM grading (coin evaluate/verify, issue #2823) is Phase 3 and stays out
of this LOCAL done-gate. No harness behaviour changed — this only gives the
done-gate a command it can run so the goal is certified complete instead of
re-stalling.

Operator Signal message (plain English, as sent)

Quick update on the "local coin benchmark harness" goal — good news, it's
actually already finished. The local benchmark and its self-improving loop are
built and every one of their built-in checks passes (7 out of 7). It kept
showing up as "stuck" only because there was no automatic test tied to it that
could confirm it was done. I've added that test now — a single command that
runs the benchmark's own self-check and passes cleanly — so the system can
confirm this goal is complete on its own and will stop reporting it as stuck.
Nothing needed from you.

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

…CLEAR-CRITERIA stops re-stalling it

The goal build-a-local-coin-benchmark-harness-and-a-self-09e65e35 stayed
Blocked cycle after cycle with why=UNCLEAR-CRITERIA ("no tracked PR/issue the
done-gate can verify"), even though the LOCAL COIN Gym harness AND its Phase-5
self-improvement loop were already built and green. The blocker was not
technical: the goal's finish condition had no definition a check could confirm,
so every OODA cycle re-observed it as unfinished and emitted NO ACTION.

Course-correction (make the done-gate machine-checkable), mirroring the #4210 /
kgpacks-parity fix:

- Specs/coin-gym-benchmark-harness.md — binds the goal's finish condition to a
  single verifiable command and enumerates the seven measurable done-criteria
  (target-loader, baseline/team runners, scorer, leaderboard-comparator,
  self-improvement-loop, contract-wiring), all asserted by the harness's own
  `coin-gym verify` self-check.
- scripts/check-coin-gym-done-gate.sh — one command the done-gate can run every
  cycle; builds coin-gym and runs `coin-gym verify`, exiting 0 only when all
  seven LOCAL criteria pass. Verified locally: 7/7 PASS, exit 0.
- docs/operations/coin-gym-goal-signal-2026-07-18.md — the plain-language
  operator/Signal notification explaining the goal is finished and will stop
  being reported as stuck (linked from mkdocs nav).

Live VM grading (coin evaluate/verify, issue #2823) is Phase 3 and stays out of
this LOCAL done-gate. No harness behaviour changed.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
rysweet added a commit that referenced this pull request Jul 18, 2026
…ne-gate triage (#4327)

Blocked-goal triage of `build-a-local-coin-benchmark-harness-and-a-self-09e65e35`.
The daemon kept parking the goal because it had no machine-checkable finish
condition it could certify against. Investigation showed the deliverable had
ALREADY shipped: issue #2713 CLOSED and PR #4171 MERGED (the `coin-gym verify`
self-check), plus supporting PRs #4208/#2740/#2763. Decision:
complete-delivered-goal — bind the goal to its closed issue and merged PR so the
done-gate certifies it automatically; do not open a third duplicate triage PR
(#4322/#4326 already open); do not escalate to a human.

Artifacts:
- docs/operations/coin-benchmark-harness-goal-signal-2026-07-18.md — durable,
  jargon-free operations record (verified-state evidence, four operator Signal
  messages, six-field OUTPUT JSON, escalate:null). Linked from operations index.
- scripts/check-coin-benchmark-harness-done-gate.sh — read-only, fail-open
  verification helper. PASS when issue #2713 CLOSED and PR #4171 MERGED; a
  genuinely failed criterion => non-zero; gh missing/unauth/offline => WARN +
  exit 0 (never a silent PASS). Optional local `coin-gym verify` is a bonus and
  is skipped (not failed) when the binary is absent or predates the verify
  subcommand. Every gh call pinned to rysweet/Simard; strict mode; no set -x.
- tests/gadugi/coin-benchmark-harness-done-gate.{sh,yaml} — outside-in TDD
  contract for both bricks (deterministic/offline via a fake gh + coin-gym).

Additive and non-breaking: no product code changed. shellcheck clean; done-gate
contract suite green (36/36); live done-gate PASSes against rysweet/Simard.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
rysweet added a commit that referenced this pull request Jul 18, 2026
…#4322)

Two pre-existing flaky lib tests intermittently failed the `pre-commit`
verify gate (which runs cargo test directly, not Python hooks), red-flagging
a cluster of otherwise merge-ready PRs (#4322, #4324, #4325, #4328). Both
flakes are on origin/main and are not PR-introduced.

- concurrent_dispatch_parallelizes_and_respects_cap: replace the brittle
  wall-clock ratio assertion (`parallel*2 <= serial`) — which jittered under
  CI scheduler contention on identical commits (#4328) — with a directional
  `parallel < serial` check. Concurrency is still proven structurally by
  run_count==N, peak_parallel>=2, and peak_serial<=1.

- meeting_turn_records_full_enriched_prompt_tokens_not_bare_objective:
  isolate the cost ledger via a new `#[cfg(test)]` thread-local
  `LedgerPathGuard` seam in cost_tracking::ledger_path() instead of mutating
  the process-global HOME. The old approach raced with any concurrent HOME
  mutator; the thread-local override cannot be observed by other threads, so
  the test drops the `cognitive_memory` serial key entirely. Both override
  branches (thread-local + SIMARD_COST_LEDGER) are strictly cfg(test)-gated,
  so the production ledger path is byte-for-byte unchanged.

Coupled observability fix: the silently-swallowed cost-write failures in the
meeting/PTY paths (base_type_copilot/mod.rs) were `eprintln!` — converted to
structured `tracing::warn!`, satisfying the no-print!/println! constraint.

Additive and test-scoped; no production behaviour change.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant