Repository navigation
v1.13.11
Changes
The game harness used to treat every failure the same way and play a random move when it ran out of retries, which made it hard to say why a game went the way it did. This PR gives each turn a full record, a deadline, a budget and a stop button. It adds a reproducible way to measure play, line-bound plan memory, sharper chess hints, resumable tournaments and opt-in experiments. Alongside that, providers gain one shared reasoning-effort option, credentials take effect without a restart, and Claude Code / Codex logins can be used as models.
Highlights
- Inspectable turns. Every move records each model call: prompt, reply, why a reply was rejected, timing and cost. Unknown prices show as a lower bound instead of $0.
- Turns can stop. Turns can have a deadline, a Stop button and a spending budget. A stopped turn never plays a move, and a reply that arrives late is ignored.
- Clear failure handling. Running out of retries can rescue (marked), forfeit or stop the game. Bad credentials stop immediately instead of producing random moves.
- Measured evaluation. Validated chess, Connect Four and Othello positions produce scorecards that keep strength, reliability and cost apart. Comparisons refuse to call scripted runs or unfair setups an improvement.
- Sharper chess hints. Warnings for mate-in-one replies, x-ray and pinned-defender exchanges, and promotions the opponent can force. Forks that are simply traded off are no longer reported.
- Memory and prompt budgets. Each side remembers its own plans on the same game line only, with checkpoints for resumed games. A prompt-size budget never drops legal moves.
- Tournaments. Side-balanced round robins resume after a stop without duplicate results, run with bounded concurrency and a shared budget, and export leaderboard eligibility flags.
- Opt-in experiments. A one-shot review of warned moves, two proposals plus a selector, and chess board images, all off by default.
- Providers. One
effortoption works across providers. Saved keys apply immediately, and local Ollama/LM Studio servers can be probed.claude-code/<model>andcodex/<model>run on a subscription login.
Technical changes
Turn records and failures (prompture/games/records.py, trace.py, base.py)
Attemptgains id, purpose, rendered prompt/schema, public or partial response, validation,FailureInfo, timing, finish reason, and per-call usage withusage_status/cost_status. Attempts are validated before they are traced.MoveDecisiongainsrecord(TurnRecord),assistanceandmodel_selected; existing positional constructors are unchanged.classify_failuremapsresilience.errors.classify_errorontoFailureKind. Status codes,Retry-Afterand cause chains are preserved, and text matching is only a last resort.Accountingcounts partial, unknown and abandoned calls.source_metareads the driver's own meta, so extraction's defaultcost: 0.0is not taken as an observed price.- New
failure_policy(rescue/forfeit/stop). Auth/configuration failures, cancellation and budget exhaustion raiseTurnFailed/TurnCancelledinstead of rescuing. fingerprint()hashes behavior settings (versions, level, concepts, memory, retry/failure policy, allowlisted options, engine identity, strategy, representation). Model IDs, names and secrets are excluded.- Traces:
JsonlTraceSink/MemoryTraceSinkwith allowlisted fields, credential redaction, a per-record byte bound, torn-line tolerantread_jsonl_trace, and a warn/raise sink-failure policy. - Extraction:
ExtractionError.raw_text/stop_reason, streampartial_text, andask_for_json(on_delta=..., should_stop=...)(StreamStoppedcloses the stream). Empty replies cut off bymax_tokenskeep their usage and say why.
Turn control (prompture/games/control.py)
deadlinecovers requests, retries and backoff. With a deadline orCancelToken, requests run on daemon worker threads. A stopped stream closes between chunks. A blocking call is markedabandoned, and its late result is only logged (late_response); no request is sent on its behalf.- Completion/cancel races are rechecked before a move is returned.
achoosecancels the turn when its task is cancelled. RetryPolicygives transient failures a separate allowance with exponential backoff, jitter and cappedRetry-After; legacy behavior is kept when it is unset.GameBudgetreserves before every request and settles afterwards (calls/tokens/cost/seconds, thread-safe, shareable). An unknown price stops cost-only budgets.TurnEventsequencing plus a bounded, delta-coalescingEventBuffer. Handler exceptions are isolated, and driver callbacks still fire.- OpenAI chat-completions and Ollama streams now close their HTTP response in
finally, so an early-closed generator releases the connection.
Evaluation (evaluation.py, fixtures.py, chess/fixtures.py)
Fixture(provenance, dev/validation/test split, initial state + move history, acceptable moves or a named evaluator, hint expectations with known-gap markers). States replay through each game'sload_state, and illegal histories raiseFixtureError.check_hintsruns with a no-call driver.evaluateruns in seeded random order with fresh players per sample and rawSamplerecords.scorecardshows numerators/denominators and amissinglist.summary_text,save_run/load_run(versioned),paired_schedule.comparewithPreregistration: paired bootstrap, reliability and cost non-inferiority gates, and refusal verdicts (not_evidence,not_comparable,not_preregistered).- 33 chess positions (mates, quiet defense, forks, exchanges, promotions, en passant, repetition, sacrifices, three saved regressions) with a
checkmateevaluator.PuzzleReport.scorecard().
Chess tactics (chess/exchange.py, chess/motifs.py)
- Bounded static exchange count over legal recaptures (pins excluded, x-rays included, promotion values), used by
losing_capture, safety, material-win andallowshints. allowsflags mate-in-one replies and opponent queen promotions the mover cannot win back, ignores forks met by an even trade, and reports when its reply cap is reached. Underpromotions are named.Motifgainsseverity,exactand anevidencewitness line;HINTS_VERSIONis 3.
Memory and prompts (memory.py, base.py)
DecisionMemory(none/previous/window) keyed byGameLine. Decisions stay pending until the line shows the move applied, orcommit()is called. Rescues carry no plan. Versioned checkpoints validate game, root and move prefix.ChessHarnessreplaces its private one-turn memory withmemory="previous".context_budgetkeeps the side, rules, rendered position, every candidate and the instructions, then adds notes urgent → warning → memory → info. When required content cannot fit, the turn fails with a configuration error.- Games gain
history(),dump_state/load_stateandsides. Connect Four and Othello set-up boards become their own root, and Othello passes are recorded as plies.
Batches (tournament.py, match.py)
Entrant/MatchContext,pair_schedule,double_round_robinovergame.sidesand start states, and a stableschedule_id.run_batch: bounded concurrency, one admission slot per provider per game, shared budget, cancel stops admission.BatchStoreJSONL with idempotent game IDs. Interrupted attempts are recorded with unknown usage before replay. A crashing entrant ends only its own game, which keeps the spend of its completed turns.GameOutcome.eligibility(),standings()over decided games, and exploratorycalibratewith minimum evidence and game/time/budget ceilings.play_matchaddsterminal_reason,forfeited_by,error,game_id,cancel, andstart_game/commithooks.
Experiments (strategies.py, chess/render.py)
Reconsider(exactly one review of a warned choice, keep/stop on review failure) andProposals(single-call workers plus a selector constrained to the proposed moves, partial-failure rules). Both run through the harness request path.representation="text+image"withRenderedImageand deterministicchess.svgboards (PNG through optionalcairosvg). Text fallbacks are recorded with a reason.- Scorecards count review fixes and harms, proposal agreement, and image turns versus fallbacks.
experiment_statusreports live-pending without live evidence.
Providers (drivers/_effort.py, infra/, coding_agent_driver.py)
options["effort"]is mapped per API family (Anthropicoutput_config.effort, OpenAI-stylereasoning_effort, OpenRouterreasoning.effort, Geminithinking_level) across sync and async drivers.- Claude streaming supports schema-enforced JSON mode (structured output or a forced
extract_jsontool) with cache breakpoints budgeted for the tool. OpenAI forwards a per-requesttimeout. CredentialStorewrites refresh changed settings and clear discovery caches in-process. Windows junctions are detected on Python < 3.12.- New
provider_keys(key catalogue and live validation, shared withprompture setup) andlocal_servers(parallel probes of Ollama/LM Studio model lists). claude-code/<model>andcodex/<model>drivers (sync/async, plugin-registered) run the signed-in CLI in an empty temp directory.costis 0 with the CLI's estimate kept inreported_cost.
Docs and examples
docs/GAMES.mdcovers the APIs and the host integration contract.examples/game_progress_example.py,game_evaluation_example.pyandgame_tournament_example.pyuse local drivers or engines unless--modelis passed.