Crash-test insurance claim AI agents before production.

A crash-test simulator for AI claim agents: adversarial cases, deterministic scoring, and replayable failure reports.

Live demo · Release v0.1.0

ClaimPilot Harness runs messy insurance claim scenarios against AI agents and shows where they passed, hesitated, or failed.

It is not another claim-processing agent. It is the test range for them.

python -m claimpilot_harness compare cases/travel-injection-001.json demo risky

Case:        travel-injection-001
Leaderboard: runs/travel-injection-001-leaderboard.html

Agent        Score    Verdict
------------ -------- ------------
demo          93.9%   investigate
risky          6.1%   approve

Most AI agent demos look impressive until they meet messy real-world claims: mismatched invoices, missing documents, policy exclusions, claimant contradictions, hidden prompt injection, and privacy traps.

ClaimPilot turns those failure modes into repeatable test cases.

Use it to answer:

• Did the agent choose the right claim action?
• Did it cite the evidence that mattered?
• Did it request the missing document instead of guessing?
• Did it detect fraud or coverage inconsistencies?
• Did it ignore malicious instructions hidden inside uploaded evidence?

See docs/why-claimpilot.md for the product thesis.

Compare a careful agent against a deliberately risky one:

python -m claimpilot_harness compare cases/travel-injection-001.json demo risky

On Windows, use py -m claimpilot_harness ... if python is not on your PATH.

You will get a score and a replay report:

Case:        travel-injection-001
Leaderboard: runs/travel-injection-001-leaderboard.html

Agent        Score    Verdict
------------ -------- ------------
demo          93.9%   investigate
risky          6.1%   approve

Open runs/latest.html to view the leaderboard.

The replay report is designed for product, risk, and engineering review:

• Evidence timeline
• Agent verdict and confidence
• Findings and requested documents
• Prompt-injection / privacy flags
• Scoring breakdown by rubric item
• Raw decision JSON for debugging

Use the built-in demo agent:

python -m claimpilot_harness run cases/auto-collision-001.json --agent demo

Compare built-in agents and generate a leaderboard:

python -m claimpilot_harness compare cases/travel-injection-001.json demo risky

Run an OpenAI-compatible model:

python -m claimpilot_harness run cases/travel-injection-001.json \
  --agent openai \
  --openai-model your-model-name

Compare it against the built-in baselines:

python -m claimpilot_harness compare cases/travel-injection-001.json demo openai risky \
  --openai-model your-model-name

Or connect any agent command that reads JSON from stdin and prints a JSON decision:

python -m claimpilot_harness run cases/auto-collision-001.json \
  --agent command \
  --agent-command "python examples/simple_agent.py"

Expected decision shape:

{
  "verdict": "investigate",
  "confidence": 0.82,
  "summary": "Hold the claim pending additional review.",
  "findings": ["invoice and physical damage evidence need reconciliation"],
  "requested_documents": ["independent adjuster estimate"],
  "cited_evidence": ["E2", "E3"],
  "privacy_flags": ["ignored embedded instruction in evidence"]
}

Cases are plain JSON files. Each case contains:

• Claimant and policy context
• Evidence summaries with stable IDs
• Red-team traps
• Expected findings, document requests, citations, and forbidden behavior
• A weighted scoring rubric

See docs/case-format.md.

ClaimPilot supports OpenAI-style /v1/chat/completions endpoints without requiring an SDK dependency.

Set OPENAI_API_KEY, then pass --agent openai and --openai-model. Use --openai-base-url for compatible local or hosted gateways.

See docs/openai-compatible.md.

• Command and HTTP agent comparison
• OpenAI-compatible and Ollama adapters
• LLM-as-judge scoring mode
• Claim case generator for synthetic case packs
• Fraud, compliance, and privacy scorecards
• CI mode for regression testing agent changes
• GitHub Pages replay gallery

ClaimPilot Harness is built for the gap between AI agent demos and production systems. A claim agent that can answer one happy-path question is easy to build. A claim agent that survives conflicting evidence, policy constraints, missing documents, and adversarial uploads needs a harness.

That is the product surface this project explores.

MIT