fix: billed-cost reconciliation + no-pause speed keepalive#44
Merged
Conversation
Two harness bugs from the v0.3.0 spread (drafts 04 + 05): Cost-estimate divergence: token-count estimates diverged up to 4x from OpenRouter actuals in both directions (reasoning-token shapes vary per provider; caching discounts unmodeled). Agents now capture the generation ID of EVERY provider call (parse retries and failed deliberations bill tokens too), the game loop drains them into CostTracker, and the run scripts reconcile against OpenRouter /api/v1/generation — the exact billed cost per call — writing billed_cost into the summary alongside the estimate. analyze_spread passes it through as real_cost_usd so future leaderboards never need hand-patching from the dashboard. Pause-on-threat freeze: RimWorld force-pauses on threat letters; with --no-pause the loop only re-asserted speed at tick boundaries, so slow models (kimi ~6 min/tick) left the game frozen for whole deliberation windows — frozen footage, wildly non-uniform game-time per tick. A keepalive task now re-asserts speed every 10s for the whole run (no-pause mode only; speed_keepalive_s=0 disables). 513 tests pass (+22 new), mypy strict clean, smoke test green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Jun 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the two remaining harness bugs from the v0.3.0 spread reconciliation (drafts 04 + 05 in
results/spread/issues_to_file/).1. Cost-estimate divergence (up to 4x vs OpenRouter actuals)
The token-count x price estimator missed in both directions on the spread (kimi 2.3x under, nemotron 4.1x under, grok 1.5x over). Root causes: per-provider reasoning-token response shapes, unmodeled caching discounts — and a genuine undercount where parse retries / failed deliberations never reached the tracker at all.
Fix — stop estimating, reconcile against ground truth:
base_role._record_completioncaptures the generation ID of every provider call (incl. parse retries) into a per-agent buffer;drain_generation_ids()hands them off.run()for late timeout completions) intoCostTracker.record_generation_id.fetch_billed_costs()queries OpenRouterGET /api/v1/generation?id=(the exact billed USD per call, semaphore-bounded, one retry for stats lag) and returns a frozenBilledCostReport. All-misses returnsNone, never a misleading $0; partial misses are flagged as a lower bound.run_scenario.py/run_benchmark.pyreconcile automatically when the base URL is OpenRouter, writebilled_costinto the summary JSON, and print it next to the estimate.analyze_spread.pypasses it through asreal_cost_usd/cost_source— same schema we hand-patched into the v0.3.0 leaderboard, so future spreads need no dashboard transcription.The estimator stays for non-OpenRouter providers and live in-run tracking.
2. Pause-on-threat freezes slow models (
--no-pause)RimWorld force-pauses on threat letters; the loop only re-asserted speed at tick boundaries, so a slow model (kimi ~6 min/tick) left the game frozen for its entire deliberation window — frozen footage and wildly non-uniform game-time per tick across models.
Fix: a
_speed_keepalivetask re-asserts speed 3 every 10s for the whole run (no-pause mode only,speed_keepalive_s=0disables, cancelled cleanly in afinally). Setting the speed is idempotent, and per-call failures are swallowed so a flaky RIMAPI moment can''t kill it.Verification
httpx.MockTransportincl. 404/connection-error/null-cost/auth-header paths, generation-ID capture/drain on the agent, keepalive loop/error-survival/cancellation)ruffclean,mypy src/strict clean,--smoke-testgreen🤖 Generated with Claude Code