test: mandatory hermetic MCP integration suite + CI gate#9
Closed
mastermanas805 wants to merge 2 commits into
Closed
test: mandatory hermetic MCP integration suite + CI gate#9mastermanas805 wants to merge 2 commits into
mastermanas805 wants to merge 2 commits into
Conversation
The delete_resource tool description said DELETE /api/v1/resources/{token};
the OpenAPI route param is :id. Renamed to {id} and noted that the path
param value is the resource token.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Makes MCP-server testing a hard CI gate. Previously `npm test` ran a shell smoke test (test.sh) against the live api.instanode.dev — not hermetic, not runnable in CI without a cluster, and easy to skip. An MCP-server change could silently break the tools agents depend on. What this adds -------------- - test/integration.test.ts — drives the REAL built server binary over the genuine MCP stdio protocol via the official SDK client. 39 tests covering all 16 registered tools: tool registry + every input schema, success responses, error envelopes (401/402/403/404/400), the multipart create_deploy upload, bearer-token auth (none/valid/bad), the full deploy lifecycle (create→get→redeploy→delete), private deploys + tier gating, and malformed-input rejection. - test/mock-api.ts — hermetic in-process http.Server mock of the agent API (the https://api.instanode.dev/openapi.json contract). The suite runs in CI with zero network, zero cluster, zero secrets. It keeps a ledger of every resource/deployment so the cleanup sweep can assert nothing leaked. - test/live-smoke.test.ts — optional, build-flagged (INSTANODE_LIVE_SMOKE=1) provision-then-teardown smoke test against a real backend; skipped by default. Deletes its resource in a finally block. Resource cleanup (mandatory) ---------------------------- Every test that creates a paid resource or a deployment tears it down. The after() hook runs a final sweep that deletes every still-live deployment + paid resource on the mock, then asserts the deletable ledger is empty. mock.close() runs in a finally so a failed assertion can't leave the server open. CI gate ------- - package.json `test` script now runs the hermetic suite via `node --test` (pretest builds server + tests). Local `npm test` == the CI gate exactly. Legacy shell smoke test moved to `test:smoke`. - .github/workflows/ci.yml runs `npm test` on every push + PR to master/main; a non-zero exit blocks the merge. - zod added as an explicit dependency (index.ts imports it directly; it was only resolving transitively via the MCP SDK). npm test: 39 passed, 0 failed (live-smoke SKIP), exit 0, fully hermetic. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Member
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
MCP-server testing was not a real gate.
npm testran a shell smoke test (test.sh) against the liveapi.instanode.dev— not hermetic, not runnable in CI without a cluster, and easy to skip. An MCP-server change could silently break the tools agents (Claude Code, Cursor, Windsurf) depend on.This makes MCP testing a mandatory CI gate:
npm testruns a hermetic integration suite on every push + PR, and a failure blocks the merge.What
test/integration.test.ts— drives the real built server binary over the genuine MCP stdio protocol via the official SDKClient. 39 tests covering all 16 registered tools:create_*requirename;create_deployadvertisestarball_base64/private/allowed_ips/resource_bindings;claim_tokenrequiresupgrade_jwt+email).create_postgres/create_cache/create_nosql/create_queue/create_storage/create_webhook— anonymous + paid tier, claim/upgrade block, S3 credential block.list_resources,delete_resource(404 + paid delete),get_api_token,claim_resource,claim_token(success + 409 conflict).create_deploy(multipart upload),get_deployment(building→running),list_deployments,redeploy,delete_deployment;resource_bindings+env_varsmerge; private deploys + tier gating.test/mock-api.ts— hermetic in-processhttp.Servermock of the agent API (theapi.instanode.dev/openapi.jsoncontract). The suite runs in CI with zero network, zero cluster, zero secrets. It keeps a ledger of every resource/deployment so the cleanup sweep can assert nothing leaked.test/live-smoke.test.ts— optional, build-flagged (INSTANODE_LIVE_SMOKE=1) provision-then-teardown smoke test against a real backend; skipped by default; deletes its resource in afinallyblock.Resource cleanup (mandatory)
Every test that creates a paid resource or deployment tears it down inline. The
after()hook runs a final sweep: deletes every still-live deployment + paid resource on the mock, then asserts the deletable ledger is empty.mock.close()runs in afinallyso a failed assertion can't leave the server open (which would hang the runner). Anonymous-tier resources are counted, not deleted — the API forbids deleting them by contract (24h TTL).CI gate
package.jsontestscript runs the hermetic suite vianode --test(pretestbuilds server + tests). Localnpm test== the CI gate exactly. Legacy shell smoke test moved totest:smoke..github/workflows/ci.ymlrunsnpm teston every push + PR tomaster/main; non-zero exit blocks merge.zodadded as an explicit dependency (index.tsimports it directly; it was only resolving transitively via the MCP SDK).Test output
npm testfrom a clean checkout (nodist/): exit 0, fully hermetic.🤖 Generated with Claude Code