groundproof

Verify agent-reported claims against ground truth.

Automated agents (LLM coding agents, bots, scripted pipelines) report outcomes: "I merged the branch", "the file now contains X", "tests pass", "the PR is closed". These reports are sometimes wrong. groundproof takes a structured claims file and checks each claim against the real system — the git repository, the filesystem, a command's exit code, an HTTP endpoint — and produces a per-claim verdict with captured evidence. It is designed to run as a post-agent gate in CI.

groundproof performs no model calls and has no AI dependencies. Every check is a direct observation of system state.

Installation

pip install groundproof            # core (pyyaml only)
pip install "groundproof[http]"    # adds httpx for http.responds

Requires Python 3.10+.

Usage

groundproof init                   # write a commented example claims.yaml
groundproof verify claims.yaml     # verify, print a table, set exit code
groundproof verify claims.yaml --report report.json --annotations
groundproof types                  # list registered claim types

Library API:

from groundproof import verify_claims

report = verify_claims("claims.yaml")
for result in report.results:
    print(result.claim.id, result.verdict.value, result.reason)
print(report.exit_code())  # 0 / 1 / 2

Claims file schema

A claims file is YAML or JSON: either a top-level list of claims, or a mapping with a single claims key. Each claim:

Key	Required	Meaning
id	yes	unique string identifying the claim
type	yes	a registered verifier name (see catalog)
params	no	mapping of verifier-specific parameters
description	no	free text carried into the report

claims:
  - id: feature-merged
    type: git.branch_merged
    description: The widget-cache branch was merged
    params:
      branch: feature/widget-cache
      into: main

Verdicts

Every claim gets exactly one of three verdicts:

• verified — the claim was checked and holds.
• refuted — the claim was checked and does not hold.
• unverifiable — the claim could not be checked: missing tool, absent repository, unreachable network, invalid parameters.

The third value exists so that "could not check" is never reported as a pass. Each verdict carries an evidence object recording what was checked, what was observed, and a timestamp.

Verifier catalog

Type	Params	Refuted when	Unverifiable when
git.commit_exists	sha, repo?	commit absent	git missing, path not a repo
git.branch_merged	branch, into, repo?	branch not an ancestor of into	git missing, ref does not exist
git.file_changed_in	path, sha, repo?	path not touched by the commit	git missing, commit absent
fs.file_exists	path	path absent	bad params
fs.file_contains	path, one of text | regex	text/regex not found, file absent	unreadable file, bad regex
fs.file_hash	path, sha256	digest mismatch, file absent	malformed digest, unreadable file
cmd.succeeds	argv, cwd?, timeout?	nonzero exit code	executable missing, timeout
test.passes	framework (pytest), node_id, cwd?, timeout?	test fails (pytest exit 1)	node id not found, collection error
http.responds	url, method?, expect_status?, expect_body_contains?, timeout?	wrong status, body text missing	httpx not installed, network error
github.pr_state	repo (owner/name), number, state (open|closed|merged)	state differs, PR not found	gh CLI missing or errored

cmd.succeeds and test.passes capture the exit code and the tail of stdout/stderr as evidence. repo and cwd default to the current working directory. Without expect_status, http.responds accepts any 2xx status.

Exit codes

Code	Default	--strict	--lenient
0	all verified	all verified	none refuted
1	any refuted	any refuted or unverifiable	any refuted
2	any unverifiable (none refuted), or claims file malformed	claims file malformed	claims file malformed

A refuted claim fails the run in every mode.

CI usage

# .github/workflows/groundproof.yml
name: groundproof
on: [pull_request]

jobs:
  verify-claims:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0   # git claims need history
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install "groundproof[http]"
      - run: groundproof verify claims.yaml --strict --annotations --report report.json
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: groundproof-report
          path: report.json

--annotations emits GitHub Actions ::error / ::warning workflow commands so refuted and unverifiable claims appear inline on the run.

Extracting claims from agent output

Agents can embed claims directly in their markdown output in a fenced block whose info string is claims:

Work complete.

```claims
- id: branch-merged
  type: git.branch_merged
  params: {branch: feature/widget-cache, into: main}
```

groundproof.extract.extract_claims(text) finds these blocks, parses them as YAML, and validates them through the normal schema. The extraction is purely mechanical text processing.

from groundproof import verify_claims
from groundproof.extract import extract_claims

claims = extract_claims(agent_markdown)
report = verify_claims(claims)

Writing a custom verifier

A verifier is a callable taking the claim's params mapping and returning a VerificationResult. Use the helpers to build verdicts and attach evidence as keyword arguments:

# myorg_pkg/verifiers.py
from groundproof.schema import VerificationResult, refuted, unverifiable, verified


def ticket_closed(params) -> VerificationResult:
    """Check that a tracker ticket is closed. Params: ticket_id."""
    ticket_id = params.get("ticket_id")
    if not isinstance(ticket_id, str):
        return unverifiable("invalid params: 'ticket_id' must be a string")
    status = my_tracker_lookup(ticket_id)   # your integration
    if status is None:
        return unverifiable(f"tracker unreachable for {ticket_id}")
    if status == "closed":
        return verified(f"{ticket_id} is closed", observed_status=status)
    return refuted(f"{ticket_id} is {status}", observed_status=status)

Register it via an entry point so groundproof discovers it when your package is installed:

# pyproject.toml of your package
[project.entry-points."groundproof.verifiers"]
"myorg.ticket_closed" = "myorg_pkg.verifiers:ticket_closed"

The claim type myorg.ticket_closed is then usable in any claims file, and appears in groundproof types. Entry points cannot shadow built-in verifier names. For programmatic use without packaging, register on a Registry instance and pass it to verify_claims(claims, registry=...).

Conventions for verifiers:

• Return unverifiable for anything that prevents the check itself (missing tool, bad params, network failure). Never raise for expected failure modes; an uncaught exception is converted to unverifiable.
• Return refuted only when the system was actually observed and the claim does not hold.
• Put the observed values in evidence so reports are auditable.

License

Apache-2.0. See LICENSE.