Contract/CI gates for the MCP tool surface and reform-input validation#42
Merged
Conversation
The hosted tool surface was only ever checked as a SUBSET (`EXPECTED_TOOLS <= names`), and the in-process check lived in a slow-marked test that PR CI skips entirely. Adding, removing or renaming a tool could therefore ship to clients with a fully green pipeline. - tests/tool_surface.py: single golden list of the 10 published tool names, shared by the in-process, stdio and live-deployment tests, with an assert_surface() helper that reports added/removed names. - tests/test_tool_surface.py: fast, PolicyEngine-free contract on exact names, exact count, no duplicates, per-tool description/schema, plus a regression guard that country/people stay schema-optional (#38). - test_remote_mcp.py / test_mcp_server.py: subset checks upgraded to exact-surface equality against the same golden list. - test_remote_mcp.py: reform-input validation (#38) now asserted AS SERVED — date-range rejection, missing country, empty reform, and a valid dated reform that must be accepted and score correctly. - test-integration.yml: the tool-surface and reform-input suites run as named hard gates before the general suite. - validate-deployment.yml: weekly Sunday deep run so the slow investment-closure and OBR bridge legs are exercised on a schedule instead of only on a manual dispatch. Also fixes a genuinely broken job: contract.yml pinned astral-sh/setup-uv@v8, but that action publishes no moving `v8` tag, so the nightly contract run had been failing in ~5s at "Set up job" since the action bump in #36. Pinned to v8.3.2. Verified: fast suite and the full live remote suite (14 passed) run locally against the deployed server. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What CI covered before
test-integration.yml— fast suite on PR/push tointegration/**.deploy-mcp.yml— deploy + remote smoke test.validate-deployment.yml— daily + post-deploy live correctness; slow legs only on manualdeepdispatch.contract.yml— nightly upstream contracts. Failing since CI hygiene: job timeouts + action version bumps #36 (see below).Gap: the tool surface was only ever checked as a subset (
EXPECTED_TOOLS <= names), and the in-process check sat in aslow-marked test that PR CI skips. A tool added, removed or renamed shipped with a green pipeline.Added
tests/tool_surface.py— golden list of the 10 published tool names, shared by the in-process, stdio and live tests.tests/test_tool_surface.py— exact names, exact count, no duplicates, per-tool description/schema, and a guard thatcountry/peoplestay schema-optional (Actionable reform-input validation for the MCP/CLI tools #38 regression).test_remote_mcp.py/test_mcp_server.pyupgraded to exact equality.test-integration.yml: both suites wired as named hard gates.validate-deployment.yml: weekly Sunday deep cron for the slow investment-closure / OBR-bridge legs.Genuine failure found
contract.ymlpinnedastral-sh/setup-uv@v8, but that action publishes no movingv8major tag (latest isv8.3.2). The nightly contract job has been dying in ~5s at "Set up job" since the action bump in #36 — e.g. run 29676226032. Pinned tov8.3.2.Verification
Fast suite green, and the full live remote suite run against the deployed Modal server: 14 passed, 2 skipped (slow). No assertions loosened or tests skipped.
🤖 Generated with Claude Code