promptcheckup 1.1.0

This release fixes three ways 1.0.x could report "better" for a prompt that wasn't: when runs failed, when the agent changed its own input, and when a cached old-prompt answer was paired with a fresh new-prompt answer. If a 1.0.x gate report listed errors, or your agent modifies the case it receives, run that gate again.

Changed

• Any failed run, under either prompt, turns a "better" verdict into "not sure yet" and says why. Failed pairs used to be dropped.
• Cached answers are reused only when the cache holds both prompts' answers for a case and sample; otherwise both run again back to back. The cache key now includes the case input.
• checkup and propose write proposals from one part of your cases and gate them on the rest. The new run.holdout setting defaults to 0.5, so a proposal needs about 40 cases before it can be tested; with fewer, the report shows it untested and doctor warns.
• The split used by checkup, propose and evolve holds out an exact number of cases, chosen by a SHA-256 hash of their IDs. Any ID now works. Splits differ from 1.0.x.
• The check-up compares with the latest snapshot, whatever its prompt, so a rollback is checked against last week.
• The check-up says "Stable" only after a clean drift comparison. The first run, partly failed runs and runs where every call failed (exit 3) have their own headlines.

Fixed

• Each agent call gets its own copy of the case input and prompts, and stored answers are copied.
• evolve no longer crashes on case IDs that aren't hexadecimal.
• @record copies the input before the call and skips duplicate IDs, so its files always load.
• Redaction now covers agent error text and answers loaded from older snapshots.
• assert_no_lookahead also changes each future number at random and removes future rows, so it catches a view that compares two future values.
• A "not sure yet" report suggests adding cases instead of running again.
• Gate reports show how often the judge gave the same answer in both orders.
• Documentation: stated the assumptions behind each simulated result, split the unreviewed-regression result by size, corrected the false "better" worst case, the privacy FAQ, the cost cap, the evolve confidence description, and several places where the design described the code wrongly. Added GEPA to CITATION.cff.

pip install --upgrade promptcheckup

promptcheckup 1.0.1

Fixed

• Workflow templates install the published package. The CI templates written by promptcheckup init now install promptcheckup>=1,<2 from PyPI instead of this repository's default branch, so your CI runs a signed release. If you ran init with 1.0.0, change the install line in .github/workflows/promptcheckup-gate.yml and promptcheckup-checkup.yml to python -m pip install "promptcheckup>=1,<2".
• README figures:
  • the false "better" rate is 2.2–2.7% on symmetric noise, and 2.0% falling to almost 0% on crash-skewed P&L;
  • ten variants tried on the same cases carry about a 22% chance of a false "better";
  • evolve keeps that chance below 2.5%.
• The integration guide installs from PyPI.

Changed

• The README is rewritten in a plainer voice. It adds a table of how many cases are needed to detect small, medium and large improvements, and a section on what the results mean in practice.

pip install --upgrade promptcheckup

promptcheckup 1.0.0

The first public release of promptcheckup: know whether a prompt change actually made your agent better.

pip install promptcheckup

Highlights

• A statistical gate:

  • a paired 95% t confidence interval with an exact Student t quantile;
  • at least 20 cases;
  • three verdicts: better, worse, or not sure yet;
  • a skew guard that refuses "better" when a few large losses hide among small gains.

• Paired replay. Every case runs through both prompts back to back, so shared noise cancels. Outputs are cached, calls are capped, and errors are reported, never dropped.

• promptcheckup gate compares the prompts in a pull request with its base branch, and a workflow template comments the report on the pull request.

• promptcheckup checkup, the weekly check-up:

  • drift against last week's snapshot, including after a prompt change, so an unreviewed merge is caught;
  • failure patterns;
  • one small proposed edit that must pass the gate.

  It never merges.

• promptcheckup evolve: repeated proposals, learned on one fixed set of cases and gated on held-out cases at a stricter confidence for every extra round. It never merges.

• A trial ledger reports how many variants were tried on the same cases and how many false "better" verdicts that many trials would produce by chance.

• promptcheckup doctor checks a project's wiring in seconds, including whether the agent really uses the prompts it is given.

• promptcheckup.testing: assert_better, assert_not_worse and assert_no_lookahead for an existing test suite.

• Judge mode: pairwise, order-swapped comparisons with a self-check against the old prompt on every run, and protection against data that tries to inject instructions.

• Agents in any language: a Python function or any command speaking JSON over stdin and stdout, with @record to capture real cases and redaction before anything is stored or sent.

Evidence

• In the gate benchmark, comparing prompts run on different weeks picked the worse prompt 41.3% of the time; paired replay picked it 0% of the time.
• In a simulated year of weekly check-ups:
  • a silent provider regression was flagged in 87.5% of incident weeks;
  • an unreviewed prompt regression was flagged in 98.5%;
  • quiet weeks raised about one false alarm per year.
• In the trading example, a win count favoured the new policy 26 to 13, but the gate said "not sure yet". Across all 2,085 ticker-days its true edge is indistinguishable from zero.

Every number is reproducible from the repository, and a test checks the README against the committed results. See the changelog for the full list.