Skip to content

v1.13.11

Choose a tag to compare

@github-actions github-actions released this 14 Oct 03:30
13e3dcd

Changes

The game harness used to treat every failure the same way and play a random move when it ran out of retries, which made it hard to say why a game went the way it did. This PR gives each turn a full record, a deadline, a budget and a stop button. It adds a reproducible way to measure play, line-bound plan memory, sharper chess hints, resumable tournaments and opt-in experiments. Alongside that, providers gain one shared reasoning-effort option, credentials take effect without a restart, and Claude Code / Codex logins can be used as models.

Highlights

  • Inspectable turns. Every move records each model call: prompt, reply, why a reply was rejected, timing and cost. Unknown prices show as a lower bound instead of $0.
  • Turns can stop. Turns can have a deadline, a Stop button and a spending budget. A stopped turn never plays a move, and a reply that arrives late is ignored.
  • Clear failure handling. Running out of retries can rescue (marked), forfeit or stop the game. Bad credentials stop immediately instead of producing random moves.
  • Measured evaluation. Validated chess, Connect Four and Othello positions produce scorecards that keep strength, reliability and cost apart. Comparisons refuse to call scripted runs or unfair setups an improvement.
  • Sharper chess hints. Warnings for mate-in-one replies, x-ray and pinned-defender exchanges, and promotions the opponent can force. Forks that are simply traded off are no longer reported.
  • Memory and prompt budgets. Each side remembers its own plans on the same game line only, with checkpoints for resumed games. A prompt-size budget never drops legal moves.
  • Tournaments. Side-balanced round robins resume after a stop without duplicate results, run with bounded concurrency and a shared budget, and export leaderboard eligibility flags.
  • Opt-in experiments. A one-shot review of warned moves, two proposals plus a selector, and chess board images, all off by default.
  • Providers. One effort option works across providers. Saved keys apply immediately, and local Ollama/LM Studio servers can be probed. claude-code/<model> and codex/<model> run on a subscription login.
Technical changes

Turn records and failures (prompture/games/records.py, trace.py, base.py)

  • Attempt gains id, purpose, rendered prompt/schema, public or partial response, validation, FailureInfo, timing, finish reason, and per-call usage with usage_status/cost_status. Attempts are validated before they are traced.
  • MoveDecision gains record (TurnRecord), assistance and model_selected; existing positional constructors are unchanged.
  • classify_failure maps resilience.errors.classify_error onto FailureKind. Status codes, Retry-After and cause chains are preserved, and text matching is only a last resort.
  • Accounting counts partial, unknown and abandoned calls. source_meta reads the driver's own meta, so extraction's default cost: 0.0 is not taken as an observed price.
  • New failure_policy (rescue/forfeit/stop). Auth/configuration failures, cancellation and budget exhaustion raise TurnFailed/TurnCancelled instead of rescuing.
  • fingerprint() hashes behavior settings (versions, level, concepts, memory, retry/failure policy, allowlisted options, engine identity, strategy, representation). Model IDs, names and secrets are excluded.
  • Traces: JsonlTraceSink/MemoryTraceSink with allowlisted fields, credential redaction, a per-record byte bound, torn-line tolerant read_jsonl_trace, and a warn/raise sink-failure policy.
  • Extraction: ExtractionError.raw_text/stop_reason, stream partial_text, and ask_for_json(on_delta=..., should_stop=...) (StreamStopped closes the stream). Empty replies cut off by max_tokens keep their usage and say why.

Turn control (prompture/games/control.py)

  • deadline covers requests, retries and backoff. With a deadline or CancelToken, requests run on daemon worker threads. A stopped stream closes between chunks. A blocking call is marked abandoned, and its late result is only logged (late_response); no request is sent on its behalf.
  • Completion/cancel races are rechecked before a move is returned. achoose cancels the turn when its task is cancelled.
  • RetryPolicy gives transient failures a separate allowance with exponential backoff, jitter and capped Retry-After; legacy behavior is kept when it is unset.
  • GameBudget reserves before every request and settles afterwards (calls/tokens/cost/seconds, thread-safe, shareable). An unknown price stops cost-only budgets.
  • TurnEvent sequencing plus a bounded, delta-coalescing EventBuffer. Handler exceptions are isolated, and driver callbacks still fire.
  • OpenAI chat-completions and Ollama streams now close their HTTP response in finally, so an early-closed generator releases the connection.

Evaluation (evaluation.py, fixtures.py, chess/fixtures.py)

  • Fixture (provenance, dev/validation/test split, initial state + move history, acceptable moves or a named evaluator, hint expectations with known-gap markers). States replay through each game's load_state, and illegal histories raise FixtureError.
  • check_hints runs with a no-call driver. evaluate runs in seeded random order with fresh players per sample and raw Sample records.
  • scorecard shows numerators/denominators and a missing list. summary_text, save_run/load_run (versioned), paired_schedule.
  • compare with Preregistration: paired bootstrap, reliability and cost non-inferiority gates, and refusal verdicts (not_evidence, not_comparable, not_preregistered).
  • 33 chess positions (mates, quiet defense, forks, exchanges, promotions, en passant, repetition, sacrifices, three saved regressions) with a checkmate evaluator. PuzzleReport.scorecard().

Chess tactics (chess/exchange.py, chess/motifs.py)

  • Bounded static exchange count over legal recaptures (pins excluded, x-rays included, promotion values), used by losing_capture, safety, material-win and allows hints.
  • allows flags mate-in-one replies and opponent queen promotions the mover cannot win back, ignores forks met by an even trade, and reports when its reply cap is reached. Underpromotions are named.
  • Motif gains severity, exact and an evidence witness line; HINTS_VERSION is 3.

Memory and prompts (memory.py, base.py)

  • DecisionMemory (none/previous/window) keyed by GameLine. Decisions stay pending until the line shows the move applied, or commit() is called. Rescues carry no plan. Versioned checkpoints validate game, root and move prefix.
  • ChessHarness replaces its private one-turn memory with memory="previous".
  • context_budget keeps the side, rules, rendered position, every candidate and the instructions, then adds notes urgent → warning → memory → info. When required content cannot fit, the turn fails with a configuration error.
  • Games gain history(), dump_state/load_state and sides. Connect Four and Othello set-up boards become their own root, and Othello passes are recorded as plies.

Batches (tournament.py, match.py)

  • Entrant/MatchContext, pair_schedule, double_round_robin over game.sides and start states, and a stable schedule_id.
  • run_batch: bounded concurrency, one admission slot per provider per game, shared budget, cancel stops admission.
  • BatchStore JSONL with idempotent game IDs. Interrupted attempts are recorded with unknown usage before replay. A crashing entrant ends only its own game, which keeps the spend of its completed turns.
  • GameOutcome.eligibility(), standings() over decided games, and exploratory calibrate with minimum evidence and game/time/budget ceilings.
  • play_match adds terminal_reason, forfeited_by, error, game_id, cancel, and start_game/commit hooks.

Experiments (strategies.py, chess/render.py)

  • Reconsider (exactly one review of a warned choice, keep/stop on review failure) and Proposals (single-call workers plus a selector constrained to the proposed moves, partial-failure rules). Both run through the harness request path.
  • representation="text+image" with RenderedImage and deterministic chess.svg boards (PNG through optional cairosvg). Text fallbacks are recorded with a reason.
  • Scorecards count review fixes and harms, proposal agreement, and image turns versus fallbacks. experiment_status reports live-pending without live evidence.

Providers (drivers/_effort.py, infra/, coding_agent_driver.py)

  • options["effort"] is mapped per API family (Anthropic output_config.effort, OpenAI-style reasoning_effort, OpenRouter reasoning.effort, Gemini thinking_level) across sync and async drivers.
  • Claude streaming supports schema-enforced JSON mode (structured output or a forced extract_json tool) with cache breakpoints budgeted for the tool. OpenAI forwards a per-request timeout.
  • CredentialStore writes refresh changed settings and clear discovery caches in-process. Windows junctions are detected on Python < 3.12.
  • New provider_keys (key catalogue and live validation, shared with prompture setup) and local_servers (parallel probes of Ollama/LM Studio model lists).
  • claude-code/<model> and codex/<model> drivers (sync/async, plugin-registered) run the signed-in CLI in an empty temp directory. cost is 0 with the CLI's estimate kept in reported_cost.

Docs and examples

  • docs/GAMES.md covers the APIs and the host integration contract.
  • examples/game_progress_example.py, game_evaluation_example.py and game_tournament_example.py use local drivers or engines unless --model is passed.

Details