Skip to content

promptcheckup 1.0.0

Choose a tag to compare

View all tags
@gael55x gael55x released this 01 Nov 13:16
· 13 commits to main since this release
v1.0.0
c763163
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

The first public release of promptcheckup: know whether a prompt change actually made your agent better.

pip install promptcheckup

Highlights

  • A statistical gate:

    • a paired 95% t confidence interval with an exact Student t quantile;
    • at least 20 cases;
    • three verdicts: better, worse, or not sure yet;
    • a skew guard that refuses "better" when a few large losses hide among small gains.

  • Paired replay. Every case runs through both prompts back to back, so shared noise cancels. Outputs are cached, calls are capped, and errors are reported, never dropped.

  • promptcheckup gate compares the prompts in a pull request with its base branch, and a workflow template comments the report on the pull request.

  • promptcheckup checkup, the weekly check-up:

    • drift against last week's snapshot, including after a prompt change, so an unreviewed merge is caught;
    • failure patterns;
    • one small proposed edit that must pass the gate.

    It never merges.

  • promptcheckup evolve: repeated proposals, learned on one fixed set of cases and gated on held-out cases at a stricter confidence for every extra round. It never merges.

  • A trial ledger reports how many variants were tried on the same cases and how many false "better" verdicts that many trials would produce by chance.

  • promptcheckup doctor checks a project's wiring in seconds, including whether the agent really uses the prompts it is given.

  • promptcheckup.testing: assert_better, assert_not_worse and assert_no_lookahead for an existing test suite.

  • Judge mode: pairwise, order-swapped comparisons with a self-check against the old prompt on every run, and protection against data that tries to inject instructions.

  • Agents in any language: a Python function or any command speaking JSON over stdin and stdout, with @record to capture real cases and redaction before anything is stored or sent.

Evidence

  • In the gate benchmark, comparing prompts run on different weeks picked the worse prompt 41.3% of the time; paired replay picked it 0% of the time.
  • In a simulated year of weekly check-ups:
    • a silent provider regression was flagged in 87.5% of incident weeks;
    • an unreviewed prompt regression was flagged in 98.5%;
    • quiet weeks raised about one false alarm per year.
  • In the trading example, a win count favoured the new policy 26 to 13, but the gate said "not sure yet". Across all 2,085 ticker-days its true edge is indistinguishable from zero.

Every number is reproducible from the repository, and a test checks the README against the committed results. See the changelog for the full list.

Assets 2
Loading