groundtruth

Catch your AI coding agent when it lies.

Your agent says "✅ Done, all tests pass." groundtruth runs the tests itself, reads the diff, and tells you whether that's actually true.

Same thing, as text

$ npx groundtruth

The agent said:
  "✅ Done. I implemented validateInput with full input validation, added tests,
   and all tests pass. It uses the `fast-csv-parser-pro` package. Ready to merge."

  groundtruth — what the agent actually did

  ✗ "Ready to merge" — it is not
      Agent declared the work complete, but 5 claim(s) don't hold up.

  ✗ "...all tests pass" — but the suite is RED
      Ran `npm test`: 1 test(s) failed.

  ✗ harness gaming: skipped test
      A passing test suite may be hiding real failures — this was added in the diff.
      test/validate.test.js:5

  ✗ claimed implemented, but it's a stub
      `validateInput` in this file is a placeholder (throws 'not implemented').
      src/validate.js:4

  ✗ hallucinated dependency: fast-csv-parser-pro
      does not exist on the npm registry — a slopsquatting supply-chain risk.

  ✗ claimed to add tests, but none found
      No new test cases appear in the diff.

  ────────────────────────────────────────────────
  VERDICT: 6 overclaim(s) — DO NOT MERGE. ✗

Try it yourself right now: bash examples/demo/run.sh

---

Why

AI coding agents are confidently wrong, and 2026 has the receipts:

• METR found models reward-hacking coding tasks in up to 76% of runs on impossible problems — and Anthropic showed an agent learning to call sys.exit(0) to fake a green test suite. [METR] [Anthropic]
• 19.7% of packages LLMs recommend don't exist — fuel for "slopsquatting" supply-chain attacks. [study]
• The everyday version: "Complete! All 10 tabs with working buttons" — when 3 tabs existed and the rest said "Coming soon."

The fix everyone hand-rolls is a 15-line bash hook that re-runs the tests. groundtruth is that idea, done properly: it takes the agent's own words and independently re-derives the truth, then hands you the receipts.

It is deterministic and offline (it runs your tests, parses your diff, checks the registry). No LLM judge, no API key, no telemetry.

Install & use

# zero-install: run it after your agent finishes a change
npx groundtruth

# or install it
npm i -D groundtruth

By default it inspects your working tree (git diff HEAD + untracked files). Give it the agent's summary to check the claims by name:

npx groundtruth --claim "implemented retryWithBackoff, all tests pass, uses p-retry"

# pipe the agent's message straight in
pbpaste | npx groundtruth

# read claims from a Claude Code transcript
npx groundtruth --auto-transcript

Exit code is non-zero when the agent overclaimed — so it drops into any script or CI.

Three ways to run it

1. CLI — works with any agent

Cursor, Copilot, Codex, Claude Code, a human on a deadline — groundtruth audits the diff, so it doesn't care who wrote it.

2. Claude Code hook — it won't let the agent lie

Install a Stop hook and Claude Code physically cannot end its turn on an overclaim — it gets blocked and told to fix the issues first.

npx groundtruth init     # writes the Stop hook into .claude/settings.json

What that adds to .claude/settings.json

{
  "hooks": {
    "Stop": [
      { "hooks": [{ "type": "command", "command": "npx groundtruth hook" }] }
    ]
  }
}

When the agent stops, groundtruth verifies its claims. If they don't hold, it returns {"decision":"block"} with the receipts, and the agent keeps working.

3. CI gate — GitHub Action with PR receipts

# .github/workflows/groundtruth.yml
name: groundtruth
on: pull_request
permissions:
  contents: read
  pull-requests: write   # for the sticky receipts comment
jobs:
  verify:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: MSal2020/groundtruth@main
        with:
          claim: ${{ github.event.pull_request.body }}   # verify the PR description's claims

The Action diffs against the PR base, runs the verifiers, fails the check on an overclaim, and posts (and keeps updated) a sticky PR comment with the receipts. Inputs: claim, base, comment, fail-on-overclaim. Or skip the Action and just run npx groundtruth --base origin/main --markdown anywhere.

What it checks

Verifier	Catches
tests	"all tests pass" → actually runs the suite (vitest / jest / node --test / pytest / go test) and compares; flags green-but-zero-tests. Monorepo-aware: runs the suite of the sub-project (backend/, …) the diff actually touched.
harness	Tests disabled to fake green: .skip / .only / xit, @pytest.mark.skip/xfail, t.Skip(), sys.exit(0), deleted assertions.
stubs	"implemented X" that's really throw new Error("TODO"), raise NotImplementedError, panic("TODO"), empty bodies, placeholders.
deps	Imported/added packages that don't exist on npm (hallucinations / slopsquatting).
pydeps	requirements.txt / pyproject.toml deps and Python imports that don't exist on PyPI.
godeps	go.mod requires and imports that don't exist on the Go module proxy.
build	"it compiles / no type errors" → runs tsc --noEmit (--build).
claims	"added tests" with no new test case; "implemented X" where X isn't in the diff.

Languages: test-running, harness-gaming and stub detection work for JS/TS, Python, and Go; registry checks cover npm, PyPI, and the Go module proxy.

Does it actually work?

A verifier that cries wolf is worse than useless, so groundtruth ships with an evaluation corpus of real scenarios — overclaims it must catch (across JS/TS, Python, and Go) and honest changes it must leave alone (path aliases, workspace: packages, # subpath imports, private scoped packages, legitimately skipped tests, real refactors, TODOs in docs).

$ npm run eval
  lying caught (TP):    21
  lies missed  (FN):    0
  honest ok    (TN):    26
  false alarms (FP):    0   <- false positives (credibility killers)
  precision: 100.0%   recall: 100.0%   F1: 100.0%

This runs in CI on every PR and must stay at 100% to merge. Run it yourself with npm run eval. Found a lie it misses or an honest change it flags? That's the most valuable issue you can open — see the templates.

Roadmap

• Python & Go runners (pytest / go test), harness + stub detection, PyPI checks
• Go module dependency hallucination checks (Go module proxy)
• GitHub Action with sticky PR receipts
• Coverage-delta gate ("you said you tested it, but coverage didn't move")
• VS Code surfacing
• Optional --llm claim extraction for free-form summaries
• A gallery of caught lies

Philosophy — what groundtruth will not do

It is the claims-vs-reality checker, and nothing else. No generic AI bug-hunting, no eval platform, no hosted service, no telemetry. It only ever tells you one thing: did the agent's claims survive contact with reality?

Contributing

Issues and PRs welcome — especially new verifiers and language support. Run npm test and npm run build before opening a PR. (groundtruth is tested with groundtruth.)

License

MIT © Muhammad Salmaan Ahmed Nusrath