agent-claimcheck 0.1.1

Packaging release. The code, benchmark and recorded results are the same as in 0.1.0.

• Published on PyPI: pip install agent-claimcheck, or uvx agent-claimcheck check example:mixed. The package README links to the tagged sources on GitHub.
• The licence is declared as the SPDX expression Apache-2.0.
• CI and the publishing workflow use current action versions.
• bench.json floats are written with 12 significant digits, so bench --from-recorded stays byte-identical across machines. The Markdown report is unchanged.

See the 0.1.0 release for what the project does and the recorded results.

agent-claimcheck 0.1.0

First release of agent-claimcheck: a library, CLI and local dashboard that checks an AI agent's success claims against the evidence in its trace (tool results and state probes) and returns verified, false_success or unverifiable with a calibrated probability. What cannot be verified goes to a human review queue.

Install / try it

uvx --from git+https://github.com/B0yko/agent-claimcheck@v0.1.0 agent-claimcheck check example:mixed
uvx --from git+https://github.com/B0yko/agent-claimcheck@v0.1.0 agent-claimcheck bench --from-recorded recorded:v0.1.0
uvx --from git+https://github.com/B0yko/agent-claimcheck@v0.1.0 agent-claimcheck serve bench:test

What ships

• agent-trace/v1 loading and validation, probe merging, label redaction, claim extraction with a negation guard.
• Detectors: declarative claim-evidence rules (booking, crm and coding packs, plus your own), a logistic-regression classifier stored as JSON, an LLM judge for any OpenAI-compatible endpoint, trust-agent and any-error baselines, and the cascade-offline and cascade ensembles, all through one shared gate with Platt calibration.
• Budget enforcement for live judge runs: a conservative cost reservation before each call, a per-run cap and a lifetime ledger cap.
• A local review dashboard (localhost only, Host/Origin checks) whose decisions are written back as agent-trace/v1 for train.
• benchmark/v1: 300 synthetic traces (180 train / 120 test), 3 domains, 7 false-success kinds, regenerated byte for byte from seed 20260924, with a dataset card and leakage audit.
• results/v0.1.0: the recorded comparison of 9 detectors (three OpenRouter judges, 1,200 calls, $0.18 total), reproducible offline with bench --from-recorded.

Headline numbers (test split, n = 120, 48 false successes)

• cascade-offline: AUROC 0.953, 90.8% decided automatically, 4/48 false successes missed, 0/72 false alarms, no API cost.
• cascade (rules, then the best judge on 30% of traces): 96.7% decided, 7/48 missed, $0.065 per 1,000 traces.
• The AUROC intervals of rules, classifier and the two stronger judges overlap; the README's Results and Findings sections report where the detectors actually differ, including every result that goes against this project's own detectors, and the limitations of a synthetic benchmark.

Apache-2.0.