test(live): cross-era conformance against public MCP servers - #261
Conversation
Pins one representative public server per protocol revision (2025-03-26 through 2026-07-28) plus pin/legacy override coverage, so negotiation regressions surface against real implementations and not only the committed fixtures. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
ClawSweeper status: review started. I am starting a fresh review of this pull request: test(live): cross-era conformance against public MCP servers This is item 1/1 in the current shard. Shard 0/1. This placeholder means the worker is alive and reading the current context. I will edit this same comment with the actual review when the claws are done clicking. Crustacean status: shell secured, claws on keyboard, evidence pebbles being sorted. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7a37cb7533
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| }).catch((error: unknown) => { | ||
| const failure = error as { stdout?: string; stderr?: string }; | ||
| return { stdout: failure.stdout ?? '', stderr: failure.stderr ?? '' }; |
There was a problem hiding this comment.
When a live CLI invocation exits non-zero or hits the 60s execFile timeout, this catch converts the failure into ordinary output and the assertions continue. That lets regressions pass whenever the failed command prints the expected substring before exiting or hanging (for example, list --verbose can emit Protocol: ... before a shutdown hang, and the tool-call assertions are loose enough to match error text), so the opt-in conformance suite no longer verifies successful command completion. Let unexpected execFile errors reject, or assert the exit status explicitly while only capturing output for diagnostics.
Useful? React with 👍 / 👎.
1 participant
Turns the ad-hoc wild-server validation done during the MCP 2.0 work into repeatable infrastructure.
tests/live/protocol-era-conformance.test.tspins one representative public server per protocol revision mcporter must interoperate with, plus pin/legacy override coverage and a real tool call on each era. Opt-in like the existing live tests (MCP_LIVE_TESTS=1), so the default gate stays network-free.Covered revisions, all verified reachable 2026-08-02: 2026-07-28 (javadocs.dev, Hugging Face, Cloudflare Docs), 2025-11-25 (Context7), 2025-06-18 (Microsoft Learn), 2025-03-26 (GitMCP).
tests/live/README.mdrecords the wider survey — 31 of 32 probed public endpoints connected — and two interop notes worth keeping: javadocs.dev is the only modern server observed returning a nonzerottlMswithcacheScope: "public", and Hugging Face and Cloudflare exposeserverInfoonly under the_metaio.modelcontextprotocol/serverInfokey, so a client reading the top-level field sees nothing.This exists because the committed fixtures can only test what we thought to implement; these tests catch drift against real implementations.
Proof: 10 tests passing against live servers.
🤖 Generated with Claude Code