Skip to content

Smoke Tests and Live Verification

Jump to bottom
Jm Rohmer edited this page Jul 18, 2026 · 2 revisions

Smoke Tests and Live Verification

This page lists the checks that catch problems unit tests can miss, especially reverse-proxy, Cloudflare, Claude.ai, ChatGPT, and AgentReady drift.

Local Release Gate

Run from the repository root:

go test ./...
go test -race ./...
go vet ./...
staticcheck ./...
govulncheck ./...
gitleaks detect --no-banner --redact --source .

Live AgentReady Gate

Run after deploy:

SMOKE_LIVE=1 ./scripts/check-agent-ready.sh
SMOKE_LIVE=1 ./scripts/smoke-agent-interop.sh

Expected:

  • OAuth authorization server metadata returns valid JSON.
  • Protected resource metadata returns valid JSON.
  • MCP server card returns valid JSON.
  • /auth.md contains the agent registration metadata.
  • Anonymous /mcp behavior does not expose write/admin tools.
  • resource is https://mcp.arleo.eu/mcp.
  • issuer is https://mcp.arleo.eu.

Manual Discovery Checks

curl -sk https://mcp.arleo.eu/.well-known/oauth-authorization-server | jq .
curl -sk https://mcp.arleo.eu/.well-known/oauth-protected-resource | jq .
curl -sk https://mcp.arleo.eu/.well-known/oauth-protected-resource/mcp | jq .
curl -sk https://mcp.arleo.eu/.well-known/mcp/server-card.json | jq .
curl -sk https://www.arleo.eu/auth.md | grep -E 'agent_auth_metadata|identity_assertion|credential_types_supported'

OAuth Authorize Check

Do not treat HTTP 302 as success by itself. Inspect the Location header:

curl -skI "https://mcp.arleo.eu/authorize?response_type=code&client_id=CLIENT_ID&redirect_uri=REDIRECT_URI&scope=write&state=test&code_challenge=CHALLENGE&code_challenge_method=S256"

(scope=content.read+content.write+site.admin still works too — accepted as legacy compatibility aliases, all normalizing to write since #450 — but scope=write is the canonical form to use going forward.)

Success redirects with:

?code=...

Failure redirects with:

?error=...

Do not paste real authorization codes or bearer tokens into issues, wiki pages, logs, or screenshots.

Claude.ai Check

After reconnecting Claude.ai:

  • The connector should complete OAuth without invalid_scope.
  • Read-only UX labels in Claude.ai do not mean anonymous access — but since #450, most read-only tools genuinely are anonymous now (see below).
  • A write-scoped connector should see the full write/build/admin tool set beyond the read surface.
  • Verify SRI integrity requires write.
  • Validate site and Validate front matter are ungated since #450 — they appear even with no Authorization header at all, not just for authenticated callers. If a smoke run shows them missing from an anonymous tools/list, that's a regression of #450, not expected behavior.

ChatGPT Check

After reconnecting ChatGPT:

  • The redirect URI must be accepted only if it matches the configured HTTPS host/path policy.
  • Unknown redirect hosts must not get the write scope.
  • A write-scoped client should see write/build tools; only a registered client with write gets them at all.

Cache and Proxy Check

If a live result contradicts the repo:

  1. Check OpenResty on the host.
  2. Check Nginx on the Hugo VM.
  3. Check Hugo static source files.
  4. Rebuild Hugo if static files changed.
  5. Purge only affected Cloudflare URLs if edge cache is stale.

Do not assume a Go server bug until the live route and static source are verified.

Clone this wiki locally