Canopy

An MCP server that exposes vehicle-diagnostic and CAN-bus domain logic as agent tools, with a LangGraph orchestration layer and a human-in-the-loop eval harness.

▶ Live showcase — replay real recorded traces: a cited answer and a grounded refusal, with the honest eval numbers. No install, no API key.

Status: Phase 4 (evals & human-in-the-loop) complete. A LangGraph agent answers natural-language diagnostics questions with validated, cited structured output — or refuses, grounded, when the connected source can't answer. Every consequential output can pass through a human-review interrupt whose structured corrections become permanent regression cases, and an LLM-judge scores traces against the same taxonomy the human uses. The design lives in docs/; the build ships one phase at a time.

Evaluation — the headline number

The LLM-judge agrees with human review on 85% of traces (n=20). Inter-rater reliability could not be measured with a panel — this is a solo project — so the same subset was scored twice, one week apart, reaching 90% self-agreement. The judge should therefore be read as approaching, not exceeding, the reliability ceiling of its ground truth. Agreement is perfect on the mechanically checkable failures — hallucinated_value and absence_as_negation, which a judge can verify against the trace — and every disagreement was an overconfident call (85%), a judgment a rubric only partially disciplines.

That paragraph is the point of Phase 4: a number, measured honestly, with its ceiling and its weak spot stated rather than hidden. The labels behind it are hand-seeded (a solo project has no panel) and firm up as real reviewed failures land — the calibration machinery is real and reproducible with no API key: uv run python scripts/calibrate.py.

Why a structured taxonomy, not thumbs-up/down? A thumbs-down says the answer was bad. A structured ErrorType says which of my defenses failed — whether a tool description needs a sentence, a refusal path didn't trigger, or a validator has a gap. The taxonomy is derived from the architecture's known weak points, so every label points at a fix.

Architecture

The central bet: the GenAI layers are independent of the data source. A normalizer sits between the data and the intelligence, so swapping OBD for raw CAN later does not require rewriting the tools, the agent, or the evals.

┌──────────────────────────────────────────────────┐
│ L6  Evals & human-in-the-loop                    │  GenAI   ✅ Phase 4
│ L5  Structured outputs & validation              │  GenAI   ✅ Phase 3
│ L4  Agent orchestration  (LangGraph)             │  GenAI  ✅←core Phase 3
│ L3  MCP server                                   │  GenAI   ✅ Phase 2
│ L2  Tool design & schemas                        │  GenAI   ✅ Phase 1
╞══════════════════════════════════════════════════╡  ← THE SEAM
│ L1b Domain logic (diagnostic rules)              │  expertise  ✅ Phase 0
│ L1a Normalizer  (SignalSample / SignalSeries)    │  contract   ✅ Phase 0
│ L0  Data access:  synthetic | OBD | CAN+DBC      │  plumbing   ✅ synthetic
└──────────────────────────────────────────────────┘

Everything above the seam must be ignorant of whether a number came from an OBD PID or a decoded CAN frame. A seam-enforcement test fails CI if that leaks.

What's built (Phases 0–4)

• The normalizer contract (model/signals.py) — SignalSample, SignalSeries, SignalSource. A time-ranged read is the general case; an OBD "value now" read is just a series of length one (is_point_read). Units always travel with values.
• Structured findings (model/findings.py) — Finding with mandatory evidence: a rule that asserts without citing samples is one the agent would launder into a hallucination.
• The data-access protocol (readers/base.py) — SignalReader with a first-class available_signals(), the mechanism by which the agent will later know what it cannot answer.
• A deterministic synthetic reader (readers/synthetic.py) — seeded waveforms for a canonical OBD signal set, with injectable known anomalies. The fixture Layers 2–6 are developed and eval'd against; no hardware required.
• The first diagnostic rule (domain/rules/correlation.py) — coolant rising while engine load is only moderate. Assumes a timeseries; degrades to a low-confidence finding on a point read.
• Four schema'd tools (tools/) — list_available_signals, summarize_session, get_signal, run_diagnostic_rules. Each is a Pydantic input schema + a description written as a prompt fragment + a handler that returns structured errors (with available_signals and a hint) instead of raising into the agent loop.
• The MCP server (mcp/server.py) — a thin stdio adapter over the tool layer. Schemas go over the wire as model_json_schema() verbatim; tool errors return isError payloads the model can recover from, while protocol errors stay JSON-RPC errors the model never sees. Reader selection is env-driven (CANOPY_SOURCE, resolved below the seam by readers/factory.py) — the server never learns which source it is serving.
• The LangGraph agent (agent/graph.py) — agent → tools → validate → refuse as a state graph. Structured output arrives through a submit_answer tool schema; validation failure is a turn (the Pydantic error is fed back with the failing field and a legal escape), capped at two retries, then degrading to a code-built honest answer rather than crashing. The iteration cap converts to a forced-answer degraded turn.
• The grounded refusal path (agent/contracts.py) — the headline behavior: a question the source cannot answer produces a Refusal naming the missing signal and what is available, filled by code from a tool result, never from model self-knowledge. A cross-validator rejects any answer citing a signal the trace never retrieved — confabulation caught mechanically.
• The eval harness & HITL (evals/) — a reviewable Trace (full tool-call record, skipped rules, outcome); a review gate built as a LangGraph interrupt whose correct verdicts mint from_review regression cases; a structured feedback taxonomy derived from the architecture's weak points; a regression runner with deterministic fixtures and hard assertions that run in CI; and a calibrated LLM-judge scoring the trace, not just the answer.

Trade-offs (honest)

• Synthetic-first, not hardware-first. If the architecture required a dongle to test, it would be wrong. Synthetic data is deterministic, so it doubles as the eval fixture. The cost: synthetic waveforms are plausible, not real — real captures arrive in Phase 5.
• A normalizer up front costs an afternoon. The payoff is that adding raw CAN later touches nothing above the seam. If it turns out to leak, that's recorded honestly in the build log — a diagnosed leak is a better story than a lucky clean run.
• OBD's coverage ceiling is a feature. OBD cannot see ADAS/camera signals. The right behavior when asked is a grounded refusal, not a guess — that refusal path is a core deliverable, not an edge case.

IP hygiene

No proprietary CAN databases, captures, or signal definitions — ever. Only public OBD-II PIDs, open DBCs, and synthetic/self-captured logs. See data/README.md for the provenance ledger.

Develop

uv sync --extra dev      # create env + install deps
uv run pytest            # run the suite, including the seam test
uv run ruff check .      # lint

Phase 0 done-signal:

from datetime import datetime, timedelta
from canopy.readers.synthetic import SyntheticReader

series = SyntheticReader(seed=42).read(
    "EngineRPM", datetime(2026, 1, 1), datetime(2026, 1, 1) + timedelta(seconds=10)
)
print(len(series.samples), series.unit, series.sample_rate_hz, series.is_point_read)

Phase 2 done-signal — a bare MCP client drives the real server as a subprocess (discovery, invocation, structured errors, clean shutdown; no LLM anywhere):

uv run python scripts/smoke_mcp.py

To explore the same server interactively, register it with any MCP client — e.g. in Claude Desktop's claude_desktop_config.json:

{
  "mcpServers": {
    "canopy": {
      "command": "/absolute/path/to/repo/.venv/bin/python",
      "args": ["-m", "canopy.mcp"],
      "env": { "CANOPY_SOURCE": "synthetic" }
    }
  }
}

The same server, zero code changes, backs the Phase 3 LangGraph agent — that is the decoupling MCP buys.

Phase 3 done-signal — ask the agent a question (needs a provider key in .env; copy .env.example). A grounded refusal on an unanswerable question is a success:

uv run python scripts/ask.py "Is the engine overheating?"
uv run python scripts/ask.py "Did the rear camera activate within 2 seconds?"   # → refusal

Phase 4 done-signal — the eval harness. Hard assertions run hermetically in the test suite (no key); the live replay and the judge run against a real model; the calibration number is reproducible from recorded labels with no key:

uv run pytest tests/test_eval_runner.py     # regression suite, scripted model, no key
uv run python scripts/calibrate.py          # the 85% / 90% agreement report
uv run python scripts/eval.py --judge       # live replay + LLM-judge (needs a key)