Repository navigation
v0.2.0
promptops eval is now a real regression gate. It runs every changed prompt over a fixed eval set, judges the outputs, compares them against main, and fails closed.
Install
pip install "git+https://github.com/vineetha00/promptops@v0.2.0"Don't pip install promptops from PyPI. That name belongs to an unrelated project, which installs its own promptops command and Python package. This project isn't on PyPI yet. A rename to promptops-git will follow in a separate release.
Fixed
- The gate failed open. If the judge call failed or no API key was set, the old code fell back to a word-count score. That fallback passed a prompt instructing the assistant to collect card numbers and CVVs, and scored it higher than the baseline. A missing field in the judge's reply was also silently scored 3.0. Now any generation, judge, credential or baseline failure exits 2 with no scores.
- The baseline could be bypassed. The old code compared versions inside the PR's own checkout. Now the baseline prompt and the eval set are read from the base branch with
git show, so a PR can't edit what it's measured against. - It measured the wrong thing. The judge used to grade how a prompt was written. Now each version answers every case in
.promptops/evals/<name>.json, and the judge scores the answers. The judge's reply format is enforced by the API, and its rubric has level anchors and few-shot calibration, including an example of a response that tries to manipulate the judge.
A change counts as a regression if any of three triggers fires
Prompt regressions are usually sparse and severe: a few inputs break, the rest are fine. A test on the mean alone misses that shape. In testing, a prompt that made the model refuse 2 of 6 requests passed the mean rule at p = 0.08.
- mean: the mean score drops more than 10% and a one-sided paired t-test gives p < 0.05
- collapse: one case drops by 2.0 or more points
- floor: one case scores 1 on any dimension where the baseline didn't (catches refusals and unsafe answers)
Before collapse or floor fires, the same outputs are judged a second time, so one flaky judgment can't fail the gate. The 2.0 cutoffs come from repeat runs of an unchanged prompt: no case's score moved more than 1.33 points between runs. Real regressions in testing moved single cases by 2.0–2.67.
Breaking changes
- Every prompt a PR changes needs an eval set at
.promptops/evals/<name>.json, merged to the base branch first. evalneeds Anthropic credentials. Fork PRs don't get repository secrets, so they fail with exit 2, and the message says this is expected.- Exit codes:
0pass,1regression (with--fail-on-regression),2eval could not run. - Default models: generator
claude-opus-5-5(override withPROMPTOPS_GEN_MODEL), judgeclaude-sonnet-5-5(override withPROMPTOPS_JUDGE_MODEL). The report lists the models that actually served the run, and the run fails if either changes partway through.
Also new
--report PATHwrites a JSON report with every case's output, scores and judge rationale.- Measured cost is roughly $0.20–0.35 per PR for a 20-case eval set.