Skip to content

Releases: tonydzi/claim-check

claim-check v1.0.1

Choose a tag to compare

@tonydzi tonydzi released this 04 Aug 17:24

Fixes a bug that only a real runner could show.

This machine has PyYAML; a fresh runner does not. So the built-in config parser — the one that actually runs in CI — was never exercised by a full local pass, and two cases failed the moment they hit ubuntu and macos:

regex: "processed (\\d+) rows"   # became a pattern hunting a literal backslash
cmd:   "echo '{\"k\": 1}'"       # the shell received \"k\" and the JSON broke

One defect underneath both: double-quoted scalars were not unescaped, so the claim failed for a reason that had nothing to do with the number. Escapes are now processed as YAML defines them, single quotes stay literal apart from '', and an escape the parser does not recognise is refused by name instead of passed through.

The blind spot was the more expensive half, so it is closed too: selftest.py now runs the entire suite twice — the second time with PyYAML blocked — and asserts both parsers agree on the awkward strings. 98 checks, both passes green on ubuntu and macos, Python 3.8 and 3.12.

v1 now points here.

claim-check v1.0.0

Choose a tag to compare

@tonydzi tonydzi released this 04 Aug 17:21

Every number you publish is a claim. Prove it in CI.

claim-check binds each figure in your README or report to the artifact it came from — a JSON file, a log, a command's output — recomputes it on every run, and fails the build naming both sides when they disagree.

- uses: Palo-Alto-AI-Research-Lab/claim-check@v1
  with:
    claims-file: .github/claims.yml
what went wrong:
  DRIFT    unique-events    doc says 700, source says 744  (REPORT.md:4)

CLAIM-CHECK: 6 claims - 4 ok - 2 drift - 0 errors -> DRIFT

In this release

  • two ways to bind a number: <!--claim:id--> markers, or the sentence itself via pattern:
  • sources: json: + path:, text: + regex:, cmd: (opt-in), value:
  • formats (comma, round:N, percent:N, any Python format spec) and tolerance — formatting differences forgiven, arithmetic not
  • --fix rewrites drifted values in place, then re-verifies from disk
  • exit-code contract 0 / 1 DRIFT / 2 ERROR, in which a source it could not read never presents as a clean result
  • workflow annotations with file and line, a job-summary table, an optional JSON report
  • 90 selftest checks, each one proven by a mutant that breaks it first

Security, because on a fork's pull request the claims file belongs to a stranger: cmd: is off unless you ask for it and refused outright on fork PRs, every config-supplied path is confined to the checkout, and a hanging command is killed after 60 seconds instead of idling the runner.

No LLM, no API key, no network, no dependencies. Python 3.8+ on the runner.