Release MCP 2.9.2: make benchmark provider setup hermetic#89
Conversation
Benchmark prepare seeded its providers from the live workspace coordination root, so running the suite started/cloned the live grepai+cgc backends and — because every worktree shares the workspace grepai workspace identity — cascaded a full re-embed across main and all worktrees, pegging the host CPU. - prepare_configured_providers no longer wires any seed source (drop the seed params and default_cgc_seed_source_coordination_root); benchmarks index their own pinned fixture cold. - grepai/cgc seed resolvers refuse a benchmark-scoped target (defense-in-depth). - carry the codex-benchmark isolation WIP: benchmark_mcp_config_overrides points the benchmarked Codex at its own MCP server; status.py compacts the CGC lastRefresh summary. - tests: benchmark seed-guard regression + hermetic prepare assertions. - bump version to 2.9.2 (pyproject, SERVER_VERSION fallback, README Status). Task 260619.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4728fa846d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| # Hermetic-cold: a benchmark indexes its own pinned fixture from scratch and never | ||
| # seeds from another coordination root. Seeding from the live workspace started the | ||
| # live backends as a clone source and cascaded a full re-embed across main + every | ||
| # worktree, so no seed options are wired here. |
There was a problem hiding this comment.
When a benchmark variant enables grepai-memory, this path now deliberately omits grepai_seed, and grepai_setup.prepare_enabled_provider skips the clone whenever grepai_seed_source_coordination_root is None; after that provider_setup only starts the watcher. The setup summary accepts the immediate watcher status as ready even while GrepAI reports indexingState: "indexing", so codex_benchmark_run can launch on a cold benchmark workspace (for example the TensorFlow case) against an empty or partial semantic index, corrupting the with-onboarding results. Please block until the benchmark-local GrepAI workspace reaches an indexed/complete state, or perform an explicit benchmark-local refresh, before returning from prepare.
Useful? React with 👍 / 👎.
1 participant
Summary
Running the Codex benchmark suite forced the main GrepAI index to rebuild and cascaded a full re-embed across every worktree provider stack, pegging the host CPU. Root cause: benchmark provider prepare seeded its grepai/cgc source from the live workspace coordination root, so preparing a case started/cloned the live backends; because worktrees share the workspace GrepAI
workspaceidentity, the disturbance cascaded to main + all worktrees.Changes
prepare_configured_providerswires no seed source — benchmarks index their own pinned fixture cold (hermetic); removeddefault_cgc_seed_source_coordination_root.benchmark-scoped target (defense-in-depth).benchmark_mcp_config_overridespoints the benchmarked Codex at its own MCP server);status.pycompacts the CGClastRefreshsummary.2.9.1→2.9.2.Verification
Full quality wrapper green: Ruff, Pyright (0 errors), pytest 620 passed / 3 skipped, CRAP
--fail-on-crap-thresholdOK.Follow-up (separate task)
CPU governance: a per-
ollamaCPU cap + embed-concurrency limit so even legitimate parallel worktrees stay bounded.🤖 Generated with Claude Code