DiffProof

Evidence, not green checks. DiffProof is a deterministic GitHub Action and Node.js CLI that asks one narrow question: would this pull request's tests detect small defects in the code it changed?

It compares a base ref with HEAD, finds changed JavaScript/TypeScript production lines, applies small mutations to those lines in a detached Git worktree, and reruns your test command. It writes both a human-readable PR report and a portable JSON evidence artifact.

Why

Coverage says code was executed. A passing test suite says the suite did not observe a failure. Neither necessarily means a test would catch an incorrect implementation. DiffProof makes that evidence visible without an LLM in the pass/fail path.

Requirements

• Node.js 20 or later
• Git 2.30 or later, with a repository containing the base ref and HEAD
• A deterministic test command safe to run repeatedly

Quick start

Clone this repository, then run from the repository you want to inspect:

node /path/to/diffproof/bin/diffproof.js \
  --base origin/main \
  --test "npm test" \
  --max-mutants 20

The command writes diffproof.json and diffproof.md in the current directory. Mutations run in a temporary detached Git worktree; the working tree is never edited.

GitHub Action

Use a pinned release tag once published:

name: DiffProof
on: pull_request
permissions:
  contents: read
  pull-requests: write
jobs:
  prove:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - uses: YOUR_ORG/diffproof@v1
        with:
          base: ${{ github.event.pull_request.base.sha }}
          test-command: npm test
          max-mutants: 20
          timeout-seconds: 300
          failure-threshold: survivors

The action uploads diffproof.json and diffproof.md, and maintains one bot comment on pull requests. For untrusted fork PRs, do not run secrets-bearing installation or test steps with elevated tokens.

Policy controls

Option	Default	Meaning
--max-mutants	20	Bound the runtime and test cost.
--timeout-seconds	300	Per baseline or mutation test timeout.
--failure-threshold baseline	baseline	Fail only if the baseline test command fails.
--failure-threshold survivors	—	Also fail if any mutation survives.
--failure-threshold score --min-score 80	—	Fail below a required evidence score.

What the score means

• Killed: the configured tests failed after DiffProof introduced a small defect in a changed line.
• Survived: the configured tests still passed. Inspect this changed behavior and its tests.
• No score: no supported mutation point was found. Treat this as missing evidence, not a pass.

Scope and limitations

Version 1 supports .js, .mjs, .cjs, .ts, .mts, .cts, and .tsx production files. It currently mutates strict equality/inequality, boolean literals, and inclusive comparisons on lines added or changed by the PR.

Mutation testing is evidence, not a correctness proof. A mutation can be equivalent to the original program, and product requirements, UI flows, performance, security, and environment behavior require additional review.

Development

npm test

The test suite includes a real temporary Git repository and verifies that DiffProof uses a worktree, limits work to diff lines, kills an intentional defect, and leaves the candidate checkout unchanged.

See CONTRIBUTING.md, SECURITY.md, and CHANGELOG.md.

License

MIT