periscope

A four-dimensional quality framework for testing MCP servers.

periscope goes beyond "does the tool return a response" — it tests whether your tools work correctly, whether an LLM actually reaches for them when it should, whether their descriptions are clear enough to be understood unambiguously, and whether they hold up under load.

Four testing dimensions:

• Functional — correctness across happy-path, boundary, invalid, and adversarial inputs
• Routing — does the LLM prefer your tools over answering from parametric knowledge?
• Clarity — are tool names, descriptions, and parameter schemas unambiguous to an LLM?
• Performance — latency percentiles and throughput under smoke, ramp, spike, and soak profiles

Works with any MCP server over both stdio and Streamable HTTP transports. Single binary, zero infrastructure — results persist to a local SQLite file.

Quick Start

From zero to your first periscope run in under 10 minutes.

1. Install the periscope binary with Go:

   go install github.com/agenthands/periscope/cmd/periscope@latest

2. Build the example MCP server (a tiny echo server used by the bundled test cases):

   go build -o echo-server ./testdata/integration/echo_server/

3. Create a minimal periscope.yaml in the current directory. Replace /path/to/echo-server with the absolute path to the binary you just built:

   version: "1.0"
   transport:
     type: stdio
     stdio:
       command: "/path/to/echo-server"  # Must be an absolute path — relative paths are rejected at config-load time.
   judge:
     provider: anthropic
     model: "claude-sonnet-4-20250514"
     apiKeyEnv: "ANTHROPIC_API_KEY"
     temperature: 0.0
     maxTokens: 2048
   embeddings:
     provider: voyage
     model: "voyage-3-lite"
     apiKeyEnv: "VOYAGE_API_KEY"
   tests:
     paths:
       - docs/examples/testcases/functional-echo.yaml
   thresholds:
     functional:
       passRate: 0.95
   reporting:
     formats: [markdown]
     outputDir: "./test-results"
     baselineDb: "./test-results/baseline.sqlite"

4. Export your API keys. periscope run in v1.0 constructs the embedder at startup regardless of --dimension, so both keys are required for every run:

   export ANTHROPIC_API_KEY='sk-ant-...'   # judge (Anthropic Messages API)
   export VOYAGE_API_KEY='pa-...'          # embeddings (required for every v1.0 run — see docs/INSTALL.md §6)

   If either key is missing, periscope fails fast at startup with environment variable <NAME> is not set and exits non-zero before any test runs. Lazy embedder construction (functional-only runs without a Voyage key) is tracked for v1.1.

5. Run the functional dimension against the echo server:

   periscope run --config periscope.yaml --dimension functional --tags smoke

   You should see output shaped like this:

   === functional	[PASS] score=1.0000
   
   Overall: PASS
   Total: 1  Passed: 1  Failed: 0  Errored: 0  Skipped: 0
   

   What this means: A line reading Overall: PASS means every gate for the selected dimension cleared its threshold, and the binary exits with code 0 so CI keeps moving. Open test-results/<RunID>.md for the full Markdown report including per-case detail and the exact threshold each gate evaluated.

The Four Dimensions

Each dimension targets a different failure mode. You can run one at a time (with --dimension), or let periscope run execute every dimension your config enables.

• Functional — Validates that each tool returns the correct output across four input categories: happy-path, boundary, invalid, and adversarial. The gate metric is pass rate (per-dimension fraction of cases that passed), and the default threshold is 0.95. Run this dimension on every PR — it is the fastest to execute and the most likely to catch regressions in your tool's own logic. Assertion modes range from exact-string matching to LLM-judge semantic comparison, so you pay only for the judgement you need.

• Routing — Measures whether an LLM actually chooses your tool when a user prompt should trigger it, as opposed to answering from its own parametric knowledge. The gate metrics are correct selection rate (did the LLM call the right tool?), parametric fallback rate (did it skip tools entirely?), parameter accuracy, and position-bias total-variation distance (does swapping tool order change the answer?). Run this dimension when you change a tool's name, description, or parameter schema, and whenever you add a tool that might compete with an existing one.

• Clarity — Evaluates whether tool names, descriptions, and parameter schemas communicate unambiguously to an LLM. Three evaluation methods are available: self-reflection (the judge rates understanding of the description), adversarial-paraphrase (judges a misleading rewrite side-by-side), and hallucination-detection (checks whether the LLM invents parameters that do not exist). The gate metrics are overall score, disambiguation rate, and hallucination rate. Run this dimension after any description rewrite and before shipping a new tool.

• Performance — Exercises your server under one of four load profiles — smoke, ramp, spike, or soak — and records latency with an HDR histogram so percentiles are accurate at p50, p95, and p99. The gate metrics are the configured SLA latencies (per-case or global), error rate, throughput (rps), and regression against the stored baseline. Run this dimension on a schedule (for example nightly) to catch performance drift before users feel it. The stdio transport is clamped to 10 virtual users so you cannot accidentally fork-bomb your local subprocess.

Installation

The simplest path is go install github.com/agenthands/periscope/cmd/periscope@latest, which drops a periscope binary into $(go env GOBIN) (or $GOPATH/bin). Prebuilt binaries for the release tags are attached to each GitHub release if you prefer not to compile. For local development, git clone the repo and run make build to produce ./periscope in the repo root.

See docs/INSTALL.md for full prereqs, verification steps, and API-key setup.

Usage

Periscope exposes 6 subcommands: run (execute tests), calibrate (tune the LLM judge against human-labeled fixtures), baseline (freeze the current run as the regression baseline), report (re-render a past run from SQLite), discover (probe an MCP server and list its tools), and history prune (retention maintenance). The common workflow:

periscope run --config periscope.yaml
periscope calibrate run --rubric clarity-self-reflection
periscope baseline --update
periscope report --format markdown

See docs/USAGE.md for every flag, expected output, YAML test case schema, and config reference.

Examples

Runnable test case YAMLs — one per dimension — live under docs/examples/testcases/: functional-echo.yaml, routing-echo.yaml, clarity-echo.yaml, and performance-echo.yaml. Each targets the bundled echo server at testdata/integration/echo_server/ so you can execute them end-to-end without pointing at a real production MCP server.

See docs/examples/testcases/ for runnable test cases and docs/examples/ci/ for GitHub Actions and GitLab CI snippets.

CI/CD Integration

periscope run --ci is threshold-gated: the binary exits 0 when every configured dimension clears its gates, and exits 1 the moment any gate fails — so your pipeline fails the same run your PR broke. JUnit XML output (reporting.formats: [junit]) populates the native Tests tab in GitHub Actions and GitLab CI, and the Markdown report doubles as a ready-to-post PR comment via $GITHUB_STEP_SUMMARY or a GitLab comment bot.

See the CI Integration section of docs/USAGE.md for full details and copy-pasteable snippets in docs/examples/ci/.

Contributing

Contributing guide (coming soon) — will land as docs/CONTRIBUTING.md in a later milestone. For now, see open issues and open a PR against main.

Built in Go. Designed for CI/CD.