progress-observability

A plugin for working with the Progress Observability Platform from your coding agent — observe, debug, cost-manage, and evaluate your AI systems, all over one read-only MCP connection.

Independent community project. Not affiliated with, or endorsed by, Progress Software. It connects to the Progress Observability Platform through the platform's public MCP API; "Progress Observability Platform" is Progress's product, referenced here only to say what these tools work with.

Everything reads through the progress-observability MCP server; the shared contract (tools, limits, and the untrusted-content rules) lives in references/mcp.md.

Skills

Skill	Command	What it does
trace-triage	/trace-triage	Root-cause a failed or slow run by walking its span tree; returns a diagnosis + a fix.
cost-report	/cost-report	Spend by model/app/day, quota burn, spike explanation, cheaper-model hypotheses.
coverage-gaps	/coverage-gaps	Finds production behaviors with no evaluation and prioritizes which judges to build.
generate-eval	/eval-from-trace, /eval-from-scratch	Research-grounded LLM-as-a-Judge evaluator prompts, optionally grounded in real traces. See skills/generate-eval.
health-check	/health-check	Read-only setup check: connection, key scope, whether traces are flowing, and instrumentation depth. Run it first.

The four workflow skills chain into one loop: trace-triage finds a failure → coverage-gaps confirms nothing measures it → generate-eval builds the judge → cost-report keeps the bill honest while you iterate. health-check sits outside the loop — a diagnostic you run first to confirm everything's wired up.

Setup

Two layers: the MCP connection (universal — any MCP-capable client can read the platform) and the workflows (the skills/commands, packaged per tool). Both use the same API key.

1. Get an MCP API key

Progress Observability → API Keys → MCP API Keys (paid plan). Metadata only scope covers health-check, trace-triage, cost-report, coverage-gaps, and from-scratch eval design; With content is needed only to read raw prompt/completion text (e.g. few-shot examples in generate-eval). Then expose it where your tool can read it:

export OBSERVABILITY_MCP_API_KEY="mcp_..."

2. Pick a setup

Option	What you get
Claude Code plugin	All five skills, their commands, and the MCP connection in one install.
VS Code / GitHub Copilot	All five skills packaged as Copilot instructions + prompts + MCP config.
A single skill	Just one skill, dropped into Claude Code — or handed to any agent as plain instructions — with the MCP server wired yourself.

Claude Code plugin — install and restart:

/plugin marketplace add observability-oss/progress-observability-plugin
/plugin install progress-observability@progress-observability

VS Code / Copilot — copy copilot/'s contents into your repo (.vscode/mcp.json, .github/copilot-instructions.md, .github/prompts/) and start the MCP server; full steps in copilot/README.md.

A single skill — see Use one skill on its own below.

Verify the connection with the curl snippet in references/mcp.md.

Usage

The commands and natural-language triggers are the same across tools (Claude Code slash commands and Copilot prompt files share names):

• /health-check — verify your setup before anything else (no arguments)
• /trace-triage <trace-id> or /trace-triage checkout-agent, timeouts, last hour
• /cost-report last 30 days, by model
• /coverage-gaps checkout-agent
• /eval-from-trace checkout-agent, wrong tool calls · /eval-from-scratch (paste a prompt)
• Or just ask in natural language — "why did this run stall?", "what's driving my spend?", "what should I evaluate next?" — the skills trigger on intent.

Use one skill on its own

Each skill is self-contained, so you don't have to take the whole plugin.

• Claude Code: copy a single skill into your skills directory — cp -r skills/generate-eval ~/.claude/skills/ (personal) or .claude/skills/ (project).
• Any tool: a skill's SKILL.md (plus any files it references, like references/mcp.md) works as standalone agent instructions.

Skills that read the platform still need the MCP server wired (step 2) and the key set. Without the plugin you won't get the slash commands — copy commands/ too, or just ask in natural language.

Notes

• The MCP server is read-only and enforces a 72-hour data window, per-tool ID caps, and rate limits (60/min metadata, 10/min content) — details in references/mcp.md.
• Nothing here writes back to the platform.
• The plugin bundle format (.claude-plugin/, slash commands) is specific to Claude Code; the copilot/ folder re-packages all five skills for VS Code / Copilot.

License

MIT — see LICENSE. Provided "as is", without warranty; the authors accept no liability for use of this software.

---

For the Progress Observability Platform.