Home

MCPipeChekr — Product Requirements Document

Version: 0.1.0-draft
Status: Draft
Author: Thomas Maerz
Last Updated: 2026-04-09
Repository: github.com/thomasmaerz/mcpipechekr (separate from emailindex)

1. Executive Summary & Problem Statement

1.1 Executive Summary

MCPipeChekr is an open-source, CLI-driven testing harness for Model Context Protocol (MCP) servers. It uses a structured multi-phase evaluation pipeline to quantify tool-call efficiency, detect behavioral regressions, and drive iterative MCP server improvement through an agentic fix loop. The primary implementation target is emailindex — a hybrid full-text and semantic vector search engine for email archives, exposed via MCP.

1.2 Problem Statement

MCP server tool calls are unreliable across models and runs. Developers observe that:

Different LLM providers select different tools for identical prompts
The same model produces inconsistent results between runs
Agents work around missing tool capabilities (e.g., iterating date windows because a sort parameter is absent), inflating token costs dramatically
Tool description quality directly determines whether an LLM selects the right tool at all — but this is invisible until something goes wrong in production

Current mitigation is entirely manual: run a prompt, observe the output, try to reason about what went wrong. There is no structured way to measure efficiency, capture regression signals, or close the loop between a detected bug and a committed fix.

MCPipeChekr solves this by providing a reproducible, multi-phase evaluation harness that:

Validates tool descriptions statically before execution (Phase 0.5)
Generates a ground-truth baseline (Phase 0)
Runs blind execution tasks and captures full traces (Phase 1)
Evaluates traces for efficiency and correctness against the baseline (Phase 2)
Drives agentic code fixes against structured bug reports (Phase 3)
Keeps a human in the loop at every review checkpoint (MVP)

2. Target Audience / Personas

2.1 Primary Persona — MCP Server Developer (Open Source)

Name: Alex
Role: Independent developer building and publishing an MCP server on GitHub
Goals:

Ship tool descriptions that work reliably across multiple LLM providers
Catch regressions before users report them
Understand the token cost of each missing feature or broken tool

Pain Points:

No structured way to test "does the model pick the right tool?"
No visibility into whether a fix actually reduced agent workarounds
Cannot compare tool behavior across providers without manual side-by-side testing

Technical Comfort: High. Comfortable in the terminal, uses opencode or Claude Code for agentic coding tasks.

2.2 Secondary Persona — MCP Server Maintainer (emailindex)

Name: Thomas
Role: Owner and primary contributor of the emailindex MCP server
Goals:

Quantify tool-call inconsistency across models
Build a repeatable test-fix-commit loop
Use structured bug reports to prioritize fixes by token cost impact

Pain Points:

Manual prompt testing does not produce comparable data across runs
No record of which tool calls were made, in what order, with which parameters
Regressions are invisible until a different model breaks on a query that used to work

Technical Comfort: Expert. Drives the reference implementation.

2.3 Tertiary Persona — Hiring Manager / TPM Evaluator

Name: Jordan
Role: Engineering Manager or TPM evaluating portfolio projects
Goals:

Assess the candidate's ability to decompose a complex technical system into a structured PRD
Evaluate systems thinking: phase decomposition, schema design, human-in-the-loop UX, NFRs

What They Need to See: A PRD that demonstrates deliberate scoping, clear acceptance criteria, structured data contracts, and honest out-of-scope decisions.

3. System Architecture Overview

3.1 Component Diagram

graph TD
    subgraph MCPipeChekr Harness
        CLI["harness.sh\n(orchestrator)"]
        P0["Phase 0\nGround Truth Generator"]
        P05["Phase 0.5\nmcp-tef\nTool Description Linter"]
        P1["Phase 1\nBlind Execution\n(opencode)"]
        P2["Phase 2\nTrace Evaluator\n(opencode)"]
        P3["Phase 3\nFix Agent\n(oh-my-openagent\nralph loop)"]
        HUMAN["Human Review\nCLI Checkpoint"]
    end

    subgraph Artifacts
        TASKS["tasks.yaml\nTask Definitions"]
        BASELINE["baseline.json\nGround Truth"]
        TRACES["traces/run-NNN.json\nSession Exports"]
        BUGRPT["bug_report.json\nStructured Bug Report"]
    end

    subgraph External
        MCP["emailindex\nMCP Server"]
        GIT["git\n(commit per fix)"]
        PROVIDERS["API Providers\n(per-phase flags)"]
    end

    CLI --> P0 --> BASELINE
    CLI --> P05
    CLI --> P1
    P1 --> TRACES
    TRACES --> P2
    BASELINE --> P2
    P2 --> BUGRPT
    BUGRPT --> HUMAN
    HUMAN -->|approve| P3
    HUMAN -->|exit| CLI
    P3 --> MCP
    P3 --> GIT
    TASKS --> P0
    TASKS --> P1
    MCP --> P1
    PROVIDERS --> P1
    PROVIDERS --> P2
    PROVIDERS --> P3

3.2 Technology Stack

Layer	Tool	Role
Execution runtime	opencode	Agent runner for Phases 0, 1, 2
Fix loop harness	oh-my-openagent	Ralph Wiggum loop for Phase 3
Tool description linting	mcp-tef	Static tool evaluation, Phase 0.5
Trace capture	`opencode session export`	Full JSON trace including tool calls and thinking blocks
Orchestration	`bash` + `tmux`	Shell loop, persistent MCP server, pane monitoring
Task definitions	YAML/JSON	Human-editable task file with expected outcomes
API providers	Configurable per-phase flags	e.g., GPT-4o (Phase 1), Claude Sonnet 4.6 (Phase 2), Minimax M2.5 (Phase 3)
Version control	`git`	One commit per fix cycle
Primary MCP server	emailindex	Reference implementation target

3.3 Runtime Topology

sequenceDiagram
    participant Human
    participant harness.sh
    participant opencode_server as opencode serve (persistent)
    participant MCP as emailindex MCP Server
    participant mcp_tef as mcp-tef

    harness.sh->>opencode_server: start (warm MCP connection)
    harness.sh->>mcp_tef: Phase 0.5 — lint tool descriptions
    mcp_tef-->>harness.sh: alignment report

    harness.sh->>opencode_server: Phase 0 — generate baseline
    opencode_server->>MCP: tool calls
    MCP-->>opencode_server: results
    opencode_server-->>harness.sh: baseline.json

    loop Until Human Exits
        harness.sh->>opencode_server: Phase 1 — blind execution
        opencode_server->>MCP: tool calls (blind)
        opencode_server-->>harness.sh: traces/run-NNN.json

        harness.sh->>opencode_server: Phase 2 — evaluate trace
        opencode_server-->>harness.sh: bug_report.json

        harness.sh->>Human: CLI checkpoint (present findings, request feedback)
        Human-->>harness.sh: approve / annotate / exit

        harness.sh->>opencode_server: Phase 3 — ralph fix loop
        opencode_server->>MCP: patch MCP server code
        opencode_server->>opencode_server: restart MCP server
        harness.sh->>harness.sh: git commit fix
    end

4. Phase Pipeline

Phase 0 — Ground Truth Generation

(Run once, or on explicit reset)

Run all tasks in tasks.yaml against the current MCP server. A separate evaluator agent judges each result as plausible or implausible. Output is stored as baseline.json containing {task_id, result, plausibility_score, timestamp}. This baseline is the correctness anchor for all subsequent Phase 2 evaluations.

Phase 0.5 — Tool Description Linting

(Run before every Phase 1)

Execute mcp-tef against the emailindex MCP server. Validates that:

Each tool description is specific enough for an LLM to select it correctly
No two tools have overlapping/conflicting descriptions
F1 scores, precision, and recall are computed per tool

Output: tef_report.json. Harness logs warnings for tools below a configurable precision threshold. Does not block execution unless a tool is flagged as "misleading" (high confidence, wrong selection).

Phase 1 — Blind Execution

(Every loop)

opencode runs each task from tasks.yaml with no failure conditions, no efficiency criteria, and no pass/fail framing visible to the agent. The agent discovers tools naturally. The full session — every tool call, every parameter, every response field touched, token count per call, and thinking blocks — is exported via opencode session export to traces/run-NNN.json.

Key constraint: The agent is explicitly not told what "optimal" looks like. Workarounds and inefficiencies must surface organically in the trace.

Phase 2 — Trace Evaluation

(Every loop, separate agent instance from Phase 1)

A different agent (different prompt, optionally different provider) ingests traces/run-NNN.json and baseline.json and evaluates against two axes:

Efficiency evaluation:

Did the agent use more tool calls than the known minimum for this task?
Did it fetch fields it never read?
Did it fall back to date-range iteration because a sort parameter was missing?
What is the token cost delta between observed and optimal call sequence?

Correctness evaluation:

Deterministic tasks: exact match against baseline result
Open-ended tasks: semantic similarity score against baseline result
Divergence above a configurable threshold is flagged for human review

Output: bug_report.json (see schema in Section 9).

Phase 3 — Agentic Fix Loop

(Every loop, after human approval)

oh-my-openagent's ralph loop ingests bug_report.json and patches the emailindex MCP server codebase. Each fix is scoped to a single bug entry (prioritized by token_cost_of_gap). The loop runs until bug_report.json items are resolved or max_iterations is reached. Each successful fix is committed: git commit --message "fix: {bug_id}". The MCP server is restarted with a health check before the next Phase 1 run.

Human Review Checkpoint

(Between Phase 2 and Phase 3, every loop)

The CLI presents:

A summary of bug_report.json findings (bug count, highest token cost gap, fix priorities)
Phase 2 evaluator narrative
A prompt asking the human to: approve, annotate (inject feedback that becomes Phase 3 context), or exit

MVP: typed CLI input. Post-MVP: web interface.

5. User Stories & Acceptance Criteria

Epic 1: Task Definition

US-1.1 As an MCP server developer, I want to define tasks in a YAML/JSON file so that I have a repeatable, version-controlled set of test queries.

Acceptance Criteria:

Tasks file supports fields: task_id, prompt, expected_tool, expected_result (optional), task_type (deterministic | open_ended)
Harness validates tasks file schema at startup and exits with a clear error if malformed
Tasks can be run as a full suite or filtered by task_id

Epic 2: Tool Description Linting (Phase 0.5)

US-2.1 As an MCP server developer, I want my tool descriptions linted before every test run so that I catch description quality issues before they contaminate trace data.

Acceptance Criteria:

mcp-tef runs automatically before Phase 1
Harness logs a warning (not a failure) for any tool with precision below a configurable threshold (default: 0.7)
Harness halts and requires human confirmation before continuing if any tool is classified as "misleading" (high confidence, wrong selection)
tef_report.json is written to the run's artifact directory

Epic 3: Ground Truth Generation (Phase 0)

US-3.1 As an MCP server developer, I want a baseline snapshot of task results so that Phase 2 can evaluate correctness against a known-good state.

Acceptance Criteria:

Phase 0 runs all tasks and stores {task_id, result, plausibility_score, timestamp} in baseline.json
A separate evaluator agent (not the Phase 1 runner) assigns plausibility scores
Phase 0 can be re-run explicitly with --regenerate-baseline flag; it does not run automatically on every loop
If no baseline exists, harness prompts the human to run Phase 0 before continuing

Epic 4: Blind Execution (Phase 1)

US-4.1 As an MCP server developer, I want agents to execute tasks without efficiency hints so that workarounds and inefficiencies are visible in the raw trace.

Acceptance Criteria:

Phase 1 prompt contains no pass/fail framing, no call-count guidance, no reference to optimal behavior
Full session exported via opencode session export to traces/run-{timestamp}.json
Trace includes: all tool calls, all parameters, all response fields, token count per call, thinking blocks
Phase 1 API provider is set via --phase1-provider flag (e.g., --phase1-provider openai/gpt-4o)

US-4.2 As an MCP server developer, I want per-phase API provider flags so that I can optimize cost and capability independently for each phase.

Acceptance Criteria:

CLI accepts --phase1-provider, --phase2-provider, --phase3-provider flags
Each flag accepts a provider/model string (e.g., anthropic/claude-sonnet-4-6, minimax/m2.5)
Provider config is written to the run's manifest for reproducibility
Missing provider flag falls back to a configurable default in config.yaml

Epic 5: Trace Evaluation (Phase 2)

US-5.1 As an MCP server developer, I want traces evaluated for efficiency so that I can see exactly where the agent over-called or under-utilized tool parameters.

Acceptance Criteria:

Phase 2 counts tool calls per task and compares against the known minimum viable call sequence
Phase 2 identifies fields fetched but never read in the trace
Phase 2 identifies workarounds (e.g., date-range iteration as a proxy for a missing sort parameter)
Each inefficiency is reported with token_cost_of_gap in the bug report

US-5.2 As an MCP server developer, I want traces evaluated for correctness against the baseline so that I can detect result regressions.

Acceptance Criteria:

Deterministic tasks: exact match comparison against baseline.json
Open-ended tasks: semantic similarity score; divergence above threshold (default: 0.2) is flagged
Correctness failures are included in bug_report.json as distinct entries from efficiency bugs

US-5.3 As an MCP server developer, I want the Phase 2 evaluator to emit a structured bug report so that Phase 3 can act on it unambiguously.

Acceptance Criteria:

bug_report.json conforms to the schema defined in Section 9
Entries are sorted by token_cost_of_gap descending (most expensive bugs first)
An empty bugs array is a valid output (clean run)

Epic 6: Human Review Checkpoint

US-6.1 As an MCP server developer, I want to review findings and approve or reject each fix loop so that I maintain control over what gets committed to the codebase.

Acceptance Criteria:

CLI presents bug report summary after Phase 2 completes
Human can type approve, annotate <free text>, or exit
annotate input is appended to the Phase 3 context before the fix run
exit terminates the harness cleanly and writes a session summary
No Phase 3 run begins without explicit approval

Epic 7: Agentic Fix Loop (Phase 3)

US-7.1 As an MCP server developer, I want an agentic coding loop to fix bugs from the report so that I don't have to manually implement every fix.

Acceptance Criteria:

oh-my-openagent ralph loop ingests bug_report.json and patches the emailindex MCP server
Fixes are prioritized by token_cost_of_gap (highest first)
Each fix is scoped to a single bug entry per iteration
Each successful fix produces a git commit: git commit --message "fix: {bug_id}"
Phase 3 API provider is set via --phase3-provider flag
MCP server health check passes before the next Phase 1 run begins

US-7.2 As an MCP server developer, I want the MCP server restarted reliably after each fix so that Phase 1 tests against the patched code, not the previous version.

Acceptance Criteria:

Harness restarts the MCP server process after Phase 3 completes
Health check (configurable, default: tool list request returning non-empty) must pass before continuing
If health check fails within a configurable timeout (default: 30s), harness halts and notifies the human

Epic 8: Observability & Run Artifacts

US-8.1 As an MCP server developer, I want all run artifacts stored in a consistent directory structure so that I can diff behavior before and after a fix.

Acceptance Criteria:

Each run produces a directory: runs/{run-id}/ containing manifest.json, tef_report.json, traces/, bug_report.json, phase2_narrative.md
manifest.json includes: run ID, timestamp, provider config per phase, task file hash, MCP server git SHA
Artifacts are not overwritten between runs

6. Out of Scope

The following are explicitly excluded from the MVP. Including them in scope discussions is a scope creep risk.

Item	Rationale
Web UI for the review checkpoint	Post-MVP. CLI is sufficient for the initial human-in-the-loop workflow.
Autonomous loop exit without human approval	Post-MVP. KPI thresholds for autonomous exit require empirical data from MVP runs to define responsibly.
Support for MCP servers other than emailindex	Post-MVP. emailindex is the reference implementation. Generalization is a future milestone.
Agent runners other than opencode	Post-MVP. opencode is the only supported Phase 1/2/3 runtime at launch.
CI/CD pipeline integration	Post-MVP. Harness is a developer-run CLI tool, not a continuous integration job.
Windows support	Out of scope. Harness is bash/tmux-based. macOS and Linux only.
Authentication / multi-user access	Not applicable. Single-developer local tool.
Cloud-hosted trace storage	Out of scope. Traces are local filesystem artifacts.

7. Non-Functional Requirements

7.1 Performance

Requirement	Target
MCP server cold start + health check	< 10 seconds
Phase 0.5 (mcp-tef lint) per run	< 60 seconds
Phase 1 per task	No hard limit; token cost captured in trace
Phase 2 evaluation per trace	< 120 seconds
Human review checkpoint response time	No timeout (human-gated)

7.2 Reliability

The opencode server (opencode serve) must persist across all phases in a single harness run. If it crashes, the harness must detect this and halt with a clear error rather than executing phases against a dead server.
MCP server restart failures must halt the loop rather than silently running Phase 1 against stale code.
All phase outputs must be written to disk before the next phase begins. Loss of a trace mid-run must not corrupt the run artifact directory.

7.3 Reproducibility

Every run must produce a manifest.json capturing: run ID, timestamp, provider/model per phase, git SHA of the MCP server, task file content hash, and harness version. A run must be fully reproducible given its manifest.
Provider flags must be stored in the manifest, not inferred from environment variables at evaluation time.

7.4 Security

API keys for LLM providers are read from environment variables or a .env file. They are never written to trace artifacts, bug reports, or git commits.
The harness does not transmit email content to third-party services beyond the configured LLM provider endpoints. Users are responsible for ensuring their provider agreements permit the data types being processed.
The git commit produced by Phase 3 must be reviewed by the human before pushing to any remote. The harness does not git push automatically.

7.5 Configurability

All thresholds, timeouts, provider defaults, and task file paths must be configurable via config.yaml.
Per-run overrides are accepted as CLI flags and take precedence over config.yaml.

8. Success Metrics / KPIs

8.1 MVP Success (Qualitative — human in the loop)

The MVP is successful when:

A full Phase 0 → 0.5 → 1 → 2 → human review → 3 → restart → 1 cycle completes without manual intervention beyond the review checkpoint
At least one real bug in the emailindex MCP server is detected by Phase 2, reported in bug_report.json, and fixed by Phase 3 with a valid git commit
The fix reduces the observed tool call count for the affected task in the subsequent Phase 1 run

8.2 Post-MVP KPI Targets

(To be calibrated from MVP run data before implementing autonomous exit)

KPI	Description	Target (TBD)
Tool call efficiency ratio	Observed calls / minimum viable calls per task	≤ 1.3 (30% overhead tolerance)
Correctness rate	% of tasks whose results match baseline within similarity threshold	≥ 95%
mcp-tef precision per tool	LLM tool selection precision per tool description	≥ 0.85
Token cost of gap (aggregate)	Sum of `token_cost_of_gap` across all open bugs	Monotonically decreasing across loops
Fix loop convergence	Number of loops to reach empty `bug_report.json`	Tracked; no hard target for MVP
Regression rate	% of fixes that re-introduce a previously closed bug	0% target

9. Data Schemas

9.1 Task Definition (`tasks.yaml`)

tasks:
  - task_id: "find-first-email-ron-chinn"
    prompt: "Find the first email mentioning Ron Chinn"
    task_type: "deterministic"
    expected_tool: "emailindex_query_email_database"
    expected_result: null  # populated after Phase 0

  - task_id: "summarize-recent-project-alpha"
    prompt: "Summarize recent emails about Project Alpha"
    task_type: "open_ended"
    expected_tool: null
    expected_result: null

9.2 Bug Report (`bug_report.json`)

{
  "run_id": "run-20260409-001",
  "phase1_provider": "openai/gpt-4o",
  "phase2_provider": "anthropic/claude-sonnet-4-6",
  "evaluated_at": "2026-04-09T14:32:00Z",
  "bugs": [
    {
      "id": "EFF-001-sort-asc",
      "type": "feature_gap",
      "tool": "emailindex_query_email_database",
      "task_id": "find-first-email-ron-chinn",
      "trigger": "sort_order=asc parameter absent",
      "observed": "Agent iterated 4 date-range windows to find oldest matching email",
      "expected": "sort_order=asc, limit=1 → single call returns chronologically first match",
      "acceptance_criteria": "Single call with sort_order=asc returns chronologically first match",
      "token_cost_observed": 3200,
      "token_cost_optimal": 400,
      "token_cost_of_gap": 2800
    },
    {
      "id": "BUG-002-fts-rank",
      "type": "bug",
      "tool": "emailindex_query_email_database",
      "task_id": "keyword-search-with-limit",
      "trigger": "exact_keywords + limit + fields (non-count)",
      "observed": "Error: no such column: fts.rank",
      "expected": "Results with projected fields returned without error",
      "acceptance_criteria": "Query returns results without error when fields parameter is specified",
      "token_cost_observed": 1800,
      "token_cost_optimal": 400,
      "token_cost_of_gap": 1400
    }
  ]
}

9.3 Baseline (`baseline.json`)

{
  "generated_at": "2026-04-09T10:00:00Z",
  "mcp_server_sha": "a1b2c3d",
  "tasks": [
    {
      "task_id": "find-first-email-ron-chinn",
      "result": { "email_id": "msg-00441", "date": "2019-03-12", "subject": "Re: intro" },
      "plausibility_score": 1.0,
      "evaluator": "anthropic/claude-sonnet-4-6"
    }
  ]
}

9.4 Run Manifest (`manifest.json`)

{
  "run_id": "run-20260409-001",
  "started_at": "2026-04-09T14:00:00Z",
  "harness_version": "0.1.0",
  "mcp_server_sha": "a1b2c3d",
  "task_file_hash": "sha256:abcdef...",
  "providers": {
    "phase0": "anthropic/claude-sonnet-4-6",
    "phase05": "mcp-tef",
    "phase1": "openai/gpt-4o",
    "phase2": "anthropic/claude-sonnet-4-6",
    "phase3": "minimax/m2.5"
  }
}

10. Documentation Requirements

10.1 `README.md` (Repository Root)

The README must:

Open with a screenshot or animated GIF of the harness running in a tmux split (Phase 1 trace streaming in one pane, harness loop progress in another)
Display stack icons for: opencode, oh-my-openagent, mcp-tef, and the MCP protocol logo
Include a one-paragraph problem statement
Include a Quick Start section (clone → configure → run) in under 10 steps
Link to the GitHub Wiki for architecture documentation
Be scannable in under 60 seconds for a developer evaluating whether to adopt the tool

10.2 GitHub Wiki

The Wiki must include the following pages:

Page	Contents
Home	Project overview, navigation index
Architecture	Full Mermaid component diagram, runtime topology sequence diagram, technology stack table
Phase Pipeline	One page per phase (0, 0.5, 1, 2, 3) with: purpose, inputs, outputs, provider flag, artifact produced
Data Schemas	All schemas from Section 9 with field descriptions
Configuration Reference	All `config.yaml` keys with types, defaults, and descriptions
CLI Reference	All flags, commands, and examples
Human Review Checkpoint	How to interpret the CLI output, approve/annotate/exit options
Adding Tasks	How to write tasks in `tasks.yaml`, task type guidance
Extending to Other MCP Servers	Post-MVP guide for adapting the harness to a different MCP server
Changelog	Version history

11. Glossary

Term	Definition
MCP	Model Context Protocol — open standard for connecting LLMs to external tools and data sources
emailindex	The primary MCP server implementation this harness targets — a hybrid full-text and semantic search engine for email archives
opencode	Open-source AI coding CLI; the agent runner for Phases 1, 2, and the underlying runtime for Phase 3
oh-my-openagent	OpenCode plugin harness providing the Ralph Wiggum loop, model fallbacks, and productivity features used in Phase 3
mcp-tef	MCP Tool Evaluation Framework by StacklokLabs — lints tool descriptions for quality and conflict (Phase 0.5)
Ralph Wiggum loop	Agentic coding pattern: repeat the same prompt in a fresh context window until a stop condition (`<promise>COMPLETE</promise>`) is met
Blind execution	Phase 1 mode: agent runs tasks with no efficiency hints, no pass/fail framing, allowing workarounds to surface naturally
token_cost_of_gap	The difference in tokens between the observed call sequence and the minimum viable call sequence — the primary prioritization signal for Phase 3
Ground truth	The baseline result set generated in Phase 0, used as the correctness anchor for Phase 2 evaluations
Trace	The full exported session JSON from an opencode run, including all tool calls, parameters, response fields, token counts, and thinking blocks
Human review checkpoint	The CLI gate between Phase 2 and Phase 3 where the human reviews findings and approves, annotates, or exits the loop

Home

MCPipeChekr — Product Requirements Document

Table of Contents

1. Executive Summary & Problem Statement

1.1 Executive Summary

1.2 Problem Statement

2. Target Audience / Personas

2.1 Primary Persona — MCP Server Developer (Open Source)

2.2 Secondary Persona — MCP Server Maintainer (emailindex)

2.3 Tertiary Persona — Hiring Manager / TPM Evaluator

3. System Architecture Overview

3.1 Component Diagram

3.2 Technology Stack

3.3 Runtime Topology

4. Phase Pipeline

Phase 0 — Ground Truth Generation

Phase 0.5 — Tool Description Linting

Phase 1 — Blind Execution

Phase 2 — Trace Evaluation

Phase 3 — Agentic Fix Loop

Human Review Checkpoint

5. User Stories & Acceptance Criteria

Epic 1: Task Definition

Epic 2: Tool Description Linting (Phase 0.5)

Epic 3: Ground Truth Generation (Phase 0)

Epic 4: Blind Execution (Phase 1)

Epic 5: Trace Evaluation (Phase 2)

Epic 6: Human Review Checkpoint

Epic 7: Agentic Fix Loop (Phase 3)

Epic 8: Observability & Run Artifacts

6. Out of Scope

7. Non-Functional Requirements

7.1 Performance

7.2 Reliability

7.3 Reproducibility

7.4 Security

7.5 Configurability

8. Success Metrics / KPIs

8.1 MVP Success (Qualitative — human in the loop)

8.2 Post-MVP KPI Targets

9. Data Schemas

9.1 Task Definition (tasks.yaml)

9.2 Bug Report (bug_report.json)

9.3 Baseline (baseline.json)

9.4 Run Manifest (manifest.json)

10. Documentation Requirements

10.1 README.md (Repository Root)

10.2 GitHub Wiki

11. Glossary

MCPipeChekr PRD

Clone this wiki locally

9.1 Task Definition (`tasks.yaml`)

9.2 Bug Report (`bug_report.json`)

9.3 Baseline (`baseline.json`)

9.4 Run Manifest (`manifest.json`)

10.1 `README.md` (Repository Root)