Repository navigation
Releases
v0.3.0: renamed to promptops-git
This release renames the package, the command and the Python import. It contains no behavior changes from v0.2.0.
Why
The name promptops on PyPI belongs to an unrelated project. That project installs its own promptops command and a top-level promptops Python package, so the two can't share one environment: pip overwrites files, and import promptops resolves to whichever was installed last. Everything that lives in a namespace shared with other projects is now renamed:
| v0.2.0 | v0.3.0 | |
|---|---|---|
| Package name | promptops |
promptops-git |
| Command | promptops |
promptops-git |
| Import | promptops |
promptops_git |
These are unchanged: the .promptops/ data directory, eval sets in .promptops/evals/, the PROMPTOPS_GEN_MODEL and PROMPTOPS_JUDGE_MODEL environment variables, and the project name.
Upgrading from 0.2.0
pip uninstall promptops
pip install "git+https://github.com/vineetha00/promptops@v0.3.0"- Your existing
.promptops/directories work as they are, with no migration needed. - CI: update your copy of
.github/workflows/prompt-regression.yml. The install line now points at@v0.3.0, and the command ispromptops-git eval .... - MCP config: change
"command": "promptops"to"command": "promptops-git". - Python imports:
from promptops import ...becomesfrom promptops_git import ....
Also
- The MCP server now reports the package version in
serverInfo. It used to report the MCP SDK's version. - The demo GIF is re-recorded with the new command.
Don't pip install promptops from PyPI; that's the unrelated project.
v0.2.0
promptops eval is now a real regression gate. It runs every changed prompt over a fixed eval set, judges the outputs, compares them against main, and fails closed.
Install
pip install "git+https://github.com/vineetha00/promptops@v0.2.0"Don't pip install promptops from PyPI. That name belongs to an unrelated project, which installs its own promptops command and Python package. This project isn't on PyPI yet. A rename to promptops-git will follow in a separate release.
Fixed
- The gate failed open. If the judge call failed or no API key was set, the old code fell back to a word-count score. That fallback passed a prompt instructing the assistant to collect card numbers and CVVs, and scored it higher than the baseline. A missing field in the judge's reply was also silently scored 3.0. Now any generation, judge, credential or baseline failure exits 2 with no scores.
- The baseline could be bypassed. The old code compared versions inside the PR's own checkout. Now the baseline prompt and the eval set are read from the base branch with
git show, so a PR can't edit what it's measured against. - It measured the wrong thing. The judge used to grade how a prompt was written. Now each version answers every case in
.promptops/evals/<name>.json, and the judge scores the answers. The judge's reply format is enforced by the API, and its rubric has level anchors and few-shot calibration, including an example of a response that tries to manipulate the judge.
A change counts as a regression if any of three triggers fires
Prompt regressions are usually sparse and severe: a few inputs break, the rest are fine. A test on the mean alone misses that shape. In testing, a prompt that made the model refuse 2 of 6 requests passed the mean rule at p = 0.08.
- mean: the mean score drops more than 10% and a one-sided paired t-test gives p < 0.05
- collapse: one case drops by 2.0 or more points
- floor: one case scores 1 on any dimension where the baseline didn't (catches refusals and unsafe answers)
Before collapse or floor fires, the same outputs are judged a second time, so one flaky judgment can't fail the gate. The 2.0 cutoffs come from repeat runs of an unchanged prompt: no case's score moved more than 1.33 points between runs. Real regressions in testing moved single cases by 2.0–2.67.
Breaking changes
- Every prompt a PR changes needs an eval set at
.promptops/evals/<name>.json, merged to the base branch first. evalneeds Anthropic credentials. Fork PRs don't get repository secrets, so they fail with exit 2, and the message says this is expected.- Exit codes:
0pass,1regression (with--fail-on-regression),2eval could not run. - Default models: generator
claude-opus-5-5(override withPROMPTOPS_GEN_MODEL), judgeclaude-sonnet-5-5(override withPROMPTOPS_JUDGE_MODEL). The report lists the models that actually served the run, and the run fails if either changes partway through.
Also new
--report PATHwrites a JSON report with every case's output, scores and judge rationale.- Measured cost is roughly $0.20–0.35 per PR for a 20-case eval set.
v0.1.1
Changes since v0.1.0
- Fix
ab-test --seed: scipy's Beta sampling draws from numpy's global RNG, not Python'srandom, so the seed wasn't actually making runs reproducible. Both RNGs are now seeded. - Add demo GIF to README showing the 60-second quickstart flow.
v0.1.0 — initial release
First release of promptops — a pip-installable, framework-agnostic, Git-native CLI that version-controls LLM prompts.
Highlights
- Git-native storage — plain
.txtfiles under.promptops/prompts/+ JSON registry - Semantic diff — sentence-transformers cosine similarity + sentence-level rich output
- Thompson Sampling A/B — Beta(α, β) posteriors per version, winner declared at p > 0.95
- LLM-as-judge eval —
claude-sonnet-4scores quality/relevance/safety, with heuristic fallback - MCP stdio server — exposes
get_active_version,list_prompts,get_metricsto any MCP client - Drop-in GitHub Actions —
prompt-regression.ymlblocks PRs whose prompts regress >10%
Install
pip install promptopsQuickstart
promptops init
promptops commit payment-agent "You are a payment assistant." --tag v1
promptops commit payment-agent "You are a helpful payment assistant." --tag v2
promptops diff payment-agent v1..v2
promptops ab-test payment-agent v1 v2 --metric quality
promptops rollback payment-agent --to v1