Skip to content

Hermetic benchmark provider setup

Choose a tag to compare

@Foxfire1st Foxfire1st released this 19 Jun 13:18
add1235

Agents Remember 2.9.2

Running the benchmark suite no longer reaches into your live workspace or worktree providers. Benchmarks now build their own isolated index from their pinned fixture, so a benchmark run can't trigger a re-index storm across your real provider stacks.

Highlights

  • Hermetic benchmark provider setup. prepare_configured_providers no longer wires a seed source — a benchmark indexes its own fixture cold instead of seeding from (and starting) the live workspace provider backends.
  • Seed guard (defense-in-depth). The GrepAI and CodeGraphContext seed resolvers now refuse a benchmark-scoped target, so a benchmark can never clone/seed from another stack even if a source is mis-wired.
  • Benchmarked agent uses its own MCP server. benchmark_mcp_config_overrides points the benchmarked Codex at the benchmark's own MCP configuration rather than inheriting the host workspace's.

Why it matters

Previously, starting the benchmark suite seeded its providers from the live workspace coordination root. Because every worktree shares the workspace's GrepAI workspace identity, that disturbance cascaded a full re-embed across main and every worktree at once — saturating CPU and making the machine unusable. Benchmarks are now isolated like a regular worktree.

Other

  • providers/status.py compacts the CodeGraphContext watcher lastRefresh summary.
  • Regression tests for the seed guard and hermetic prepare.

Changes: #89