test: replay MCP continuity transactions#3064
Merged
Merged
Conversation
Problem: the continuity checker only classified observations supplied by a caller, so it could not prove a real MCP client could discover, interrupt, and resume an exhaustive query.\n\nWhat changed: add a deterministic synthetic action corpus and a stdio MCP client/server replay. The runner discovers the canonical query route, records a bounded first page as the cancellation point, resumes through fresh server processes, checks the independent known-answer fixture, and can persist the receipt transcript.\n\nCompatibility: the existing observation evaluator remains available; --live-mcp adds the real execution mode.\n\nCo-Authored-By: Codex <noreply@openai.com>
|
Warning Review limit reached
Next review available in: 26 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (5)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4ba34cd30f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+158
to
+159
| if page.get("query_ref") != first.get("query_ref") or page.get("result_ref") != first.get("result_ref"): | ||
| raise RuntimeError("cold MCP resume changed transaction identity") |
There was a problem hiding this comment.
If every page omits or nulls both query_ref and result_ref, these comparisons evaluate as None == None, so the replay can still classify as pass and persist a receipt with null transaction identities. Validate that both refs on the first page are non-empty strings before using equality to prove that a cold resume preserved the transaction.
Useful? React with 👍 / 👎.
| }, | ||
| ] | ||
| pages = [first] | ||
| while isinstance(continuation, str): |
There was a problem hiding this comment.
When a regressed server returns an endless sequence of distinct continuation tokens, or a cycle longer than one token, this loop never reaches evaluate_replay; the adjacent equality check only detects a token repeated immediately. Track all seen tokens and enforce scenario.budget.max_calls while walking so the non-progressing-continuation scenario produces a failure instead of hanging the integration test or CLI indefinitely.
Useful? React with 👍 / 👎.
| "refs": refs, | ||
| "calls": len(pages), | ||
| "page_bytes": max(page_sizes, default=0), | ||
| "cancel_seconds": 0.0, |
There was a problem hiding this comment.
When stopping the first stdio server at the page boundary takes longer than the scenario's five-second cancellation budget, the replay still reports 0.0 and passes because no observed timing is supplied to the classifier. Record the actual page-boundary teardown/cancellation duration rather than making the max_cancel_seconds check unreachable for this live replay.
Useful? React with 👍 / 👎.
Sinity
added a commit
that referenced
this pull request
Jul 18, 2026
Problem polylogue-t8t needs a durable black-box spec for seven operator continuity jobs plus the parallel-Claude incident replay, graded by an independent oracle rather than a caller-supplied observation. PR #3060 (GPT-Pro lin-02 r01) rewrote the scaffold to run through the real Polylogue API facade, but never proved the MCP transport, exact-count completeness, or the original incident curriculum grading (AC 3-5 partial/remaining per the bead's own AC matrix). PR #3064 separately added one MCP-stdio scenario ("mcp-query-transaction") on top of the old dataclass scaffold, incrementally, without adopting the ScenarioSpec-based rewrite. Solution Reconciles GPT-Pro lin-02 revision r02 (a full replacement package, not a patch on r01) against current master, which had drifted from both packages' common base commit via #3064's incremental scenario: - polylogue/product/continuity_scenarios.py: ContinuityScenarioSpec now subclasses the existing scenarios.NamedScenarioSource seam. All eight jobs (seven operator jobs + parallel-Claude incident) declare typed discovery requirements, coverage inventories, accepted plan signatures, exact item-identity paths, stop conditions, and call/page/aggregate/elapsed/cancellation-grace budgets. - devtools/continuity_replay.py: default replay path launches the production read-role MCP server as a child process and drives it over the official MCP SDK/stdio JSON-RPC, recording discovery receipts before executing any scenario. Every aggregate-capable terminal unit gets a separate `| count` probe cross-checked against enumerated identities, rejecting page-local/wrapper-rewritten totals. - The parallel incident now carries a six-case sanitized curriculum (candidate-list, operator-phrase, lexical-proxy, hidden-grammar, and two correctly-formulated-but-failed-execution cases) planted separately from its expected grades, plus six executable mutation families (lost continuation state, capped pseudo-total, identical- call topology replay, hidden discovery, missing coverage, unreasonable-query classification). - polylogue/mcp/server_prompts.py: fixes two shipped prompt recipes that advertised parser-invalid `query_units` expressions (`unacknowledged_failures` embedded `since` inside an action predicate; `sessions_touching_file` used a bare `repo:` file predicate instead of `session.repo:`). tests/unit/mcp/ test_prompt_query_parity.py parses every shipped recipe against the production parser and discovered tool schema so this can't regress. - tests/infra/continuity.py / continuity_mutations.py / tests/data/ continuity/catalog.json: schema-v2 corpus/oracle separation — `seed_continuity_archive()` builds real archive tiers from planted corpus facts only; the oracle section is never derived from a production route. Reconciliation performed by hand: r02's PATCH.diff (generated against the same pre-#3064 base commit as r01) applied cleanly to continuity_scenarios.py by context-matching around #3064's inserted "mcp-query-transaction" scenario block, stranding it as a dead old-signature `_scenario(...)` call incompatible with the new constructor (7 positional args against a 3-positional/keyword-only signature) — this is removed here since r02's all-scenario MCP-stdio replay, exact-count probes, and mutation matrix supersede what that single incremental scenario proved. devtools/continuity_replay.py and the two continuity test files conflicted directly against #3064's additions during a 3-way apply; resolved by taking r02's full replacement content, since both r01 and r02 replace this architecture wholesale and #3064 built incrementally on the version being replaced. This does not close polylogue-t8t. AC 6 (z9gh.7 live-archive/cold-model terminal gate) remains open; the catalog is corpus-parameterized and ready for a live oracle but no live archive was accessed. Verification - devtools test tests/unit/product/test_continuity_scenarios.py tests/unit/mcp/test_prompt_query_parity.py tests/integration/test_continuity_replay.py -> 20 passed - devtools test tests/unit/mcp/test_server_surfaces.py -> 83 passed (no MCP tool-surface regression from the prompt-recipe fix) - ruff format --check / ruff check / mypy --strict on all 9 changed/added files -> clean - devtools render all --check -> exit 0, no modules added, no topology-projection regen needed Ref polylogue-t8t
Sinity
added a commit
that referenced
this pull request
Jul 18, 2026
…3060) ## Summary Revision r02 of the `polylogue-t8t` continuity replay slice (supersedes the r01 content previously on this branch). Replaces the surviving `ContinuityScenarioSpec` dataclass draft and the caller-supplied-observation-only `devtools/continuity_replay.py` with an executable declaration catalog that drives the real production MCP server over stdio JSON-RPC for all eight scenarios (the seven `polylogue-t8t` operator jobs plus the parallel-Claude incident), with exact-count pagination proofs, a six-case incident curriculum, and six executable mutation families. ## Problem r01 (the version previously on this branch) executed scenarios through the `Polylogue` API facade directly, not through MCP — it proved query/parser/storage behavior but not MCP discovery, transport, or response-budget behavior, and left AC items 3-5 (transcript/receipt classification, original-incident curriculum grading, and the full mutation-family set) partial. Separately, PR #3064 (`test: replay MCP continuity transactions`, merged) added one incremental MCP-stdio scenario (`mcp-query-transaction`) directly onto the still-live dataclass scaffold — narrower in scope (one scenario, not the full catalog) and architecturally incompatible with the ScenarioSpec-based rewrite both r01 and r02 make. ## Solution - `polylogue/product/continuity_scenarios.py`: `ContinuityScenarioSpec` subclasses the existing `scenarios.NamedScenarioSource` seam. Eight declarations carry typed discovery requirements, coverage inventories, accepted plan signatures, exact item-identity paths, stop conditions, and call/page/aggregate/elapsed/cancellation-grace budgets. - `devtools/continuity_replay.py`: default replay path launches the production read-role MCP server as a child process via the official MCP SDK/stdio JSON-RPC, records discovery receipts (protocol version, tool/resource schemas) before running any scenario, and cross-checks a separate `| count` probe against enumerated item identities for every aggregate-capable terminal unit — rejecting page-local or wrapper-rewritten totals. - The parallel-incident scenario adds a six-case sanitized curriculum (candidate-list-oversized, wrong-corpus-assumption, weak-lexical-proxy, product-induced-hidden-grammar, and two correctly-formulated-but-execution-failed cases) planted separately from its expected grades, plus six executable mutation families (lost continuation state, capped pseudo-total, identical-call topology replay, hidden discovery, missing source coverage, unreasonable-query classification). - `polylogue/mcp/server_prompts.py`: fixes two shipped prompt recipes that advertised parser-invalid `query_units` expressions — `unacknowledged_failures` embedded `since` inside an action predicate instead of as the top-level argument; `sessions_touching_file` used a bare `repo:` file predicate instead of session-scoped `session.repo:`. New `tests/unit/mcp/test_prompt_query_parity.py` parses every shipped `query_units` recipe against the production parser and discovered tool schema so this can't regress silently. - `tests/infra/continuity.py` / `continuity_mutations.py` / `tests/data/continuity/catalog.json`: schema-v2 corpus/oracle separation. `seed_continuity_archive()` builds real archive tiers from planted corpus facts only; `validate_continuity_population()` does a direct SQLite census with no query/MCP/replay involvement; the independent oracle section is never derived from a production route. ### Reconciliation performed by hand Both r01 and r02 packets target base commit `536a53e` (`fix(repair): harden raw authority convergence #3046`), pre-dating #3064. Applying r02's patch to current master (`git apply --3way`) applied `continuity_scenarios.py` "cleanly" by context-matching around #3064's inserted `mcp-query-transaction` scenario block — but that left the block behind as a dead call to the *old* 7-positional-argument `_scenario()` helper, incompatible with the rewritten 3-positional/keyword-only signature (`TypeError: _scenario() takes 3 positional arguments but 7 were given` at import time). Removed that stranded block: r02's all-scenario MCP-stdio replay, exact-count probes, and mutation matrix supersede what that single incremental scenario proved (an MCP transaction surviving one client-side pause/resume). `devtools/continuity_replay.py` and the two continuity test files conflicted directly against #3064's additions during the 3-way apply; resolved by taking r02's full replacement content in each case, since both revisions replace this architecture wholesale and #3064 built incrementally on the version being replaced. `tests/data/continuity/incident.json` (the pre-existing, now fully unreferenced legacy fixture, including the row #3064 added to it) is left in place unchanged, matching both packets' own decision not to delete it without separate certification. ## `polylogue-t8t` AC matrix (bead's own numbering) | # | Criterion | Status | | - | --- | --- | | 1 | Seven jobs + parallel-Claude variant | **Satisfied** — eight declarations, all pass through MCP stdio JSON-RPC. | | 2 | Wording, facts/coverage, independent target, discovery, refs, plan equivalence, paging/cancel/resource bounds, stops | **Satisfied as declaration/catalog contract.** | | 3 | Baseline transcripts/server receipts classify every failure layer | **Partial.** Discovery/call/page receipts and mutations distinguish source-coverage, discovery, execution, projection, reasoning; no cold external-model transcript, no runner-issued MCP cancellation transcript. | | 4 | Grade the original incident calls with the exposed curriculum | **Satisfied synthetically** — six raw-feature cases planted separately from six expected grades; the prompt-parity fix preserves the product-induced sessions-only classification. | | 5 | Mutation fixtures cover every named recovery/failure class | **Satisfied** — all six execute and fail in the intended class. | | 6 | `z9gh.7` consumes this catalog as sole terminal gate | **Ready, not executed** — corpus is parameterized and CLI documented; the authorized live-archive/cold-model/effect-evidence/SLO gate remains `polylogue-z9gh.7` scope. | This PR does not close `polylogue-t8t`; item 6 remains `polylogue-z9gh.7`'s scope. ## Verification - `devtools test tests/unit/product/test_continuity_scenarios.py tests/unit/mcp/test_prompt_query_parity.py tests/integration/test_continuity_replay.py` → **20 passed**. - `devtools test tests/unit/mcp/test_server_surfaces.py` → **83 passed** (no MCP tool-surface regression from the prompt-recipe fix). - `ruff format --check`, `ruff check`, `mypy --strict` on all 9 changed/added files → clean. - `devtools render all --check` → exit 0 (no new `polylogue/` module; no topology-projection regen needed). - Pre-push quick gate: 16/16 steps passed (format/lint/mypy/render-all/topology/layering/closure-matrix/schema/manifests/ci-workflows/doc-commands/docs-coverage/test-infra-currency/test-clock-hygiene/pytest-timeout-overrides/degrade-loudly). Ref polylogue-t8t --------- Co-authored-by: Codex <noreply@openai.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a deterministic terminal continuity replay that drives Polylogue through
an actual local MCP stdio client/server exchange.
Problem
The continuity replay command previously graded only observations supplied by
a caller. That could validate an invented transcript without proving that MCP
discovery, pagination, interruption, and recovery work through the registered
transport.
Solution
mcp-query-transactionknown-answer scenario.query_unitsand itscapability resource through the MCP protocol, and collect a bounded first
page.
opaque continuation, then resume remaining pages through fresh stdio MCP
server processes.
evidence, cancellation point, and cold-resume pages.
rejection, and CLI receipt persistence.
This deliberately does not modify aliases, raw-authority repair, rebuild
transactions, or Beads state.
Verification
devtools test tests/unit/product/test_continuity_scenarios.py tests/integration/test_continuity_replay.py— 8 passed.devtools verify --quick— success.Remaining scope
2qx.2still needs full Claude artifact admission/provenance anddegraded-state proof.
1vpm.6.2still needs observed git/GitHub/Beads/verification effect adapters and reconciliation proof; both remain
prerequisites before claiming the wider
z9gh.7closure.