Skip to content

Releases

v0.3.0: renamed to promptops-git

Choose a tag to compare

@vineetha00 vineetha00 released this 21 Oct 05:25

This release renames the package, the command and the Python import. It contains no behavior changes from v0.2.0.

Why

The name promptops on PyPI belongs to an unrelated project. That project installs its own promptops command and a top-level promptops Python package, so the two can't share one environment: pip overwrites files, and import promptops resolves to whichever was installed last. Everything that lives in a namespace shared with other projects is now renamed:

v0.2.0 v0.3.0
Package name promptops promptops-git
Command promptops promptops-git
Import promptops promptops_git

These are unchanged: the .promptops/ data directory, eval sets in .promptops/evals/, the PROMPTOPS_GEN_MODEL and PROMPTOPS_JUDGE_MODEL environment variables, and the project name.

Upgrading from 0.2.0

pip uninstall promptops
pip install "git+https://github.com/vineetha00/promptops@v0.3.0"
  • Your existing .promptops/ directories work as they are, with no migration needed.
  • CI: update your copy of .github/workflows/prompt-regression.yml. The install line now points at @v0.3.0, and the command is promptops-git eval ....
  • MCP config: change "command": "promptops" to "command": "promptops-git".
  • Python imports: from promptops import ... becomes from promptops_git import ....

Also

  • The MCP server now reports the package version in serverInfo. It used to report the MCP SDK's version.
  • The demo GIF is re-recorded with the new command.

Don't pip install promptops from PyPI; that's the unrelated project.

v0.2.0

Choose a tag to compare

@vineetha00 vineetha00 released this 21 Oct 02:50

promptops eval is now a real regression gate. It runs every changed prompt over a fixed eval set, judges the outputs, compares them against main, and fails closed.

Install

pip install "git+https://github.com/vineetha00/promptops@v0.2.0"

Don't pip install promptops from PyPI. That name belongs to an unrelated project, which installs its own promptops command and Python package. This project isn't on PyPI yet. A rename to promptops-git will follow in a separate release.

Fixed

  • The gate failed open. If the judge call failed or no API key was set, the old code fell back to a word-count score. That fallback passed a prompt instructing the assistant to collect card numbers and CVVs, and scored it higher than the baseline. A missing field in the judge's reply was also silently scored 3.0. Now any generation, judge, credential or baseline failure exits 2 with no scores.
  • The baseline could be bypassed. The old code compared versions inside the PR's own checkout. Now the baseline prompt and the eval set are read from the base branch with git show, so a PR can't edit what it's measured against.
  • It measured the wrong thing. The judge used to grade how a prompt was written. Now each version answers every case in .promptops/evals/<name>.json, and the judge scores the answers. The judge's reply format is enforced by the API, and its rubric has level anchors and few-shot calibration, including an example of a response that tries to manipulate the judge.

A change counts as a regression if any of three triggers fires

Prompt regressions are usually sparse and severe: a few inputs break, the rest are fine. A test on the mean alone misses that shape. In testing, a prompt that made the model refuse 2 of 6 requests passed the mean rule at p = 0.08.

  • mean: the mean score drops more than 10% and a one-sided paired t-test gives p < 0.05
  • collapse: one case drops by 2.0 or more points
  • floor: one case scores 1 on any dimension where the baseline didn't (catches refusals and unsafe answers)

Before collapse or floor fires, the same outputs are judged a second time, so one flaky judgment can't fail the gate. The 2.0 cutoffs come from repeat runs of an unchanged prompt: no case's score moved more than 1.33 points between runs. Real regressions in testing moved single cases by 2.0–2.67.

Breaking changes

  • Every prompt a PR changes needs an eval set at .promptops/evals/<name>.json, merged to the base branch first.
  • eval needs Anthropic credentials. Fork PRs don't get repository secrets, so they fail with exit 2, and the message says this is expected.
  • Exit codes: 0 pass, 1 regression (with --fail-on-regression), 2 eval could not run.
  • Default models: generator claude-opus-5-5 (override with PROMPTOPS_GEN_MODEL), judge claude-sonnet-5-5 (override with PROMPTOPS_JUDGE_MODEL). The report lists the models that actually served the run, and the run fails if either changes partway through.

Also new

  • --report PATH writes a JSON report with every case's output, scores and judge rationale.
  • Measured cost is roughly $0.20–0.35 per PR for a 20-case eval set.

v0.1.1

Choose a tag to compare

@vineetha00 vineetha00 released this 08 Jul 04:04

Changes since v0.1.0

  • Fix ab-test --seed: scipy's Beta sampling draws from numpy's global RNG, not Python's random, so the seed wasn't actually making runs reproducible. Both RNGs are now seeded.
  • Add demo GIF to README showing the 60-second quickstart flow.

v0.1.0 — initial release

Choose a tag to compare

@vineetha00 vineetha00 released this 01 Jun 00:55

First release of promptops — a pip-installable, framework-agnostic, Git-native CLI that version-controls LLM prompts.

Highlights

  • Git-native storage — plain .txt files under .promptops/prompts/ + JSON registry
  • Semantic diff — sentence-transformers cosine similarity + sentence-level rich output
  • Thompson Sampling A/B — Beta(α, β) posteriors per version, winner declared at p > 0.95
  • LLM-as-judge eval — claude-sonnet-4 scores quality/relevance/safety, with heuristic fallback
  • MCP stdio server — exposes get_active_version, list_prompts, get_metrics to any MCP client
  • Drop-in GitHub Actions — prompt-regression.yml blocks PRs whose prompts regress >10%

Install

pip install promptops

Quickstart

promptops init
promptops commit payment-agent "You are a payment assistant." --tag v1
promptops commit payment-agent "You are a helpful payment assistant." --tag v2
promptops diff payment-agent v1..v2
promptops ab-test payment-agent v1 v2 --metric quality
promptops rollback payment-agent --to v1