Skip to content

test: mandatory hermetic MCP integration suite + CI gate#9

Closed
mastermanas805 wants to merge 2 commits into
masterfrom
test/mcp-integration-suite
Closed

test: mandatory hermetic MCP integration suite + CI gate#9
mastermanas805 wants to merge 2 commits into
masterfrom
test/mcp-integration-suite

Conversation

@mastermanas805
Copy link
Copy Markdown
Member

Why

MCP-server testing was not a real gate. npm test ran a shell smoke test (test.sh) against the live api.instanode.dev — not hermetic, not runnable in CI without a cluster, and easy to skip. An MCP-server change could silently break the tools agents (Claude Code, Cursor, Windsurf) depend on.

This makes MCP testing a mandatory CI gate: npm test runs a hermetic integration suite on every push + PR, and a failure blocks the merge.

What

  • test/integration.test.ts — drives the real built server binary over the genuine MCP stdio protocol via the official SDK Client. 39 tests covering all 16 registered tools:
    • Tool registry: exactly the 16 contract tools, no dead ones; every tool's input schema asserted (create_* require name; create_deploy advertises tarball_base64/private/allowed_ips/resource_bindings; claim_token requires upgrade_jwt+email).
    • create_postgres / create_cache / create_nosql / create_queue / create_storage / create_webhook — anonymous + paid tier, claim/upgrade block, S3 credential block.
    • Auth handling — no token / valid token / bad (revoked) token → 401 with dashboard CTA.
    • list_resources, delete_resource (404 + paid delete), get_api_token, claim_resource, claim_token (success + 409 conflict).
    • Deployment lifecycle — create_deploy (multipart upload), get_deployment (building→running), list_deployments, redeploy, delete_deployment; resource_bindings+env_vars merge; private deploys + tier gating.
    • Malformed input — non-string tarball, out-of-range port, empty id, unknown tool name.
  • test/mock-api.ts — hermetic in-process http.Server mock of the agent API (the api.instanode.dev/openapi.json contract). The suite runs in CI with zero network, zero cluster, zero secrets. It keeps a ledger of every resource/deployment so the cleanup sweep can assert nothing leaked.
  • test/live-smoke.test.ts — optional, build-flagged (INSTANODE_LIVE_SMOKE=1) provision-then-teardown smoke test against a real backend; skipped by default; deletes its resource in a finally block.

Resource cleanup (mandatory)

Every test that creates a paid resource or deployment tears it down inline. The after() hook runs a final sweep: deletes every still-live deployment + paid resource on the mock, then asserts the deletable ledger is empty. mock.close() runs in a finally so a failed assertion can't leave the server open (which would hang the runner). Anonymous-tier resources are counted, not deleted — the API forbids deleting them by contract (24h TTL).

CI gate

  • package.json test script runs the hermetic suite via node --test (pretest builds server + tests). Local npm test == the CI gate exactly. Legacy shell smoke test moved to test:smoke.
  • .github/workflows/ci.yml runs npm test on every push + PR to master/main; non-zero exit blocks merge.
  • zod added as an explicit dependency (index.ts imports it directly; it was only resolving transitively via the MCP SDK).

Test output

ℹ tests 39
ℹ pass 39
ℹ fail 0
ℹ duration_ms 10806

npm test from a clean checkout (no dist/): exit 0, fully hermetic.

🤖 Generated with Claude Code

mastermanas805 and others added 2 commits May 17, 2026 09:29
The delete_resource tool description said DELETE /api/v1/resources/{token};
the OpenAPI route param is :id. Renamed to {id} and noted that the path
param value is the resource token.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Makes MCP-server testing a hard CI gate. Previously `npm test` ran a
shell smoke test (test.sh) against the live api.instanode.dev — not
hermetic, not runnable in CI without a cluster, and easy to skip. An
MCP-server change could silently break the tools agents depend on.

What this adds
--------------
- test/integration.test.ts — drives the REAL built server binary over
  the genuine MCP stdio protocol via the official SDK client. 39 tests
  covering all 16 registered tools: tool registry + every input schema,
  success responses, error envelopes (401/402/403/404/400), the
  multipart create_deploy upload, bearer-token auth (none/valid/bad),
  the full deploy lifecycle (create→get→redeploy→delete), private
  deploys + tier gating, and malformed-input rejection.
- test/mock-api.ts — hermetic in-process http.Server mock of the agent
  API (the https://api.instanode.dev/openapi.json contract). The suite
  runs in CI with zero network, zero cluster, zero secrets. It keeps a
  ledger of every resource/deployment so the cleanup sweep can assert
  nothing leaked.
- test/live-smoke.test.ts — optional, build-flagged (INSTANODE_LIVE_SMOKE=1)
  provision-then-teardown smoke test against a real backend; skipped by
  default. Deletes its resource in a finally block.

Resource cleanup (mandatory)
----------------------------
Every test that creates a paid resource or a deployment tears it down.
The after() hook runs a final sweep that deletes every still-live
deployment + paid resource on the mock, then asserts the deletable
ledger is empty. mock.close() runs in a finally so a failed assertion
can't leave the server open.

CI gate
-------
- package.json `test` script now runs the hermetic suite via
  `node --test` (pretest builds server + tests). Local `npm test` ==
  the CI gate exactly. Legacy shell smoke test moved to `test:smoke`.
- .github/workflows/ci.yml runs `npm test` on every push + PR to
  master/main; a non-zero exit blocks the merge.
- zod added as an explicit dependency (index.ts imports it directly;
  it was only resolving transitively via the MCP SDK).

npm test: 39 passed, 0 failed (live-smoke SKIP), exit 0, fully hermetic.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@mastermanas805
Copy link
Copy Markdown
Member Author

Closing as superseded. The integration test suite from this PR was incorporated into PR #10 (which itself was superseded by #12). The hermetic test suite is now on master via #12 (merged 2026-05-21). Nothing here is missing from master.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant