Skip to content

fix: billed-cost reconciliation + no-pause speed keepalive#44

Merged
jkbennitt merged 1 commit into
masterfrom
fix/billed-cost-and-speed-keepalive
Jun 12, 2026
Merged

fix: billed-cost reconciliation + no-pause speed keepalive#44
jkbennitt merged 1 commit into
masterfrom
fix/billed-cost-and-speed-keepalive

Conversation

@jkbennitt

Copy link
Copy Markdown
Member

Fixes the two remaining harness bugs from the v0.3.0 spread reconciliation (drafts 04 + 05 in results/spread/issues_to_file/).

1. Cost-estimate divergence (up to 4x vs OpenRouter actuals)

The token-count x price estimator missed in both directions on the spread (kimi 2.3x under, nemotron 4.1x under, grok 1.5x over). Root causes: per-provider reasoning-token response shapes, unmodeled caching discounts — and a genuine undercount where parse retries / failed deliberations never reached the tracker at all.

Fix — stop estimating, reconcile against ground truth:

  • base_role._record_completion captures the generation ID of every provider call (incl. parse retries) into a per-agent buffer; drain_generation_ids() hands them off.
  • The game loop drains all agents per tick (plus a final drain in run() for late timeout completions) into CostTracker.record_generation_id.
  • fetch_billed_costs() queries OpenRouter GET /api/v1/generation?id= (the exact billed USD per call, semaphore-bounded, one retry for stats lag) and returns a frozen BilledCostReport. All-misses returns None, never a misleading $0; partial misses are flagged as a lower bound.
  • run_scenario.py / run_benchmark.py reconcile automatically when the base URL is OpenRouter, write billed_cost into the summary JSON, and print it next to the estimate.
  • analyze_spread.py passes it through as real_cost_usd/cost_source — same schema we hand-patched into the v0.3.0 leaderboard, so future spreads need no dashboard transcription.

The estimator stays for non-OpenRouter providers and live in-run tracking.

2. Pause-on-threat freezes slow models (--no-pause)

RimWorld force-pauses on threat letters; the loop only re-asserted speed at tick boundaries, so a slow model (kimi ~6 min/tick) left the game frozen for its entire deliberation window — frozen footage and wildly non-uniform game-time per tick across models.

Fix: a _speed_keepalive task re-asserts speed 3 every 10s for the whole run (no-pause mode only, speed_keepalive_s=0 disables, cancelled cleanly in a finally). Setting the speed is idempotent, and per-call failures are swallowed so a flaky RIMAPI moment can''t kill it.

Verification

  • 513 tests pass (+22 new: billed-cost fetch via httpx.MockTransport incl. 404/connection-error/null-cost/auth-header paths, generation-ID capture/drain on the agent, keepalive loop/error-survival/cancellation)
  • ruff clean, mypy src/ strict clean, --smoke-test green
  • Live-game behavior (does speed=3 actually clear a threat force-pause?) to be validated on the next live run, per the usual caveat

🤖 Generated with Claude Code

Two harness bugs from the v0.3.0 spread (drafts 04 + 05):

Cost-estimate divergence: token-count estimates diverged up to 4x from
OpenRouter actuals in both directions (reasoning-token shapes vary per
provider; caching discounts unmodeled). Agents now capture the generation
ID of EVERY provider call (parse retries and failed deliberations bill
tokens too), the game loop drains them into CostTracker, and the run
scripts reconcile against OpenRouter /api/v1/generation — the exact billed
cost per call — writing billed_cost into the summary alongside the
estimate. analyze_spread passes it through as real_cost_usd so future
leaderboards never need hand-patching from the dashboard.

Pause-on-threat freeze: RimWorld force-pauses on threat letters; with
--no-pause the loop only re-asserted speed at tick boundaries, so slow
models (kimi ~6 min/tick) left the game frozen for whole deliberation
windows — frozen footage, wildly non-uniform game-time per tick. A
keepalive task now re-asserts speed every 10s for the whole run
(no-pause mode only; speed_keepalive_s=0 disables).

513 tests pass (+22 new), mypy strict clean, smoke test green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@jkbennitt jkbennitt merged commit 209b275 into master Jun 12, 2026
3 checks passed
@jkbennitt jkbennitt deleted the fix/billed-cost-and-speed-keepalive branch June 12, 2026 16:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant