Statistically honest regression gates for RAG systems in CI.
EvalFlow decides whether an LLM or RAG change is safe to merge. It runs your golden dataset enough times to separate real regressions from sampling noise, checks the retrieval layer and the generation layer separately so a broken retriever cannot hide behind a fluent answer, and reports a pass/fail your team can leave blocking.
The full design is in docs/design.md.
Working today (
pip install -e ., thenevalflow run): dataset and config loading, the deterministic mock provider, an OpenAI provider (pip install evalflow[openai], optional and lazily imported), BM25 retrieval, the per-stage execution plan, two-stage aggregation with canonical precision, retrieval metrics (recall@k, MRR), generation metrics (token-overlap similarity, exact match), system metrics (latency, error rate, cost, tokens), single-run confidence intervals, the paired-bootstrap regression gate withdetection_effectsensitivity and survivorship accounting, an off-by-default response cache keyed on repeat identity so it can never collapse a sample distribution,compare,report(offline re-render of a saved artifact), and run-artifact directories. The console block below is a live run; the flagship examples/support-assistant and the smaller examples/llm-quickstart both run offline against the mock, and both work unmodified against a real model by swappingprovider.typetoopenaiand settingOPENAI_API_KEY.Not yet built: embedding-based similarity, sequential testing, flake quarantine,
evalflow power, rerankers, and vector retrievers.
promptfoo already covers broad, CI-friendly LLM evaluation across 60+ providers, and does it well. EvalFlow is not trying to compete on breadth. It targets two specific ways a quality gate returns the wrong verdict — and both of them end with the gate being turned off or trusted when it should not be.
False alarms from noise. LLM output is non-deterministic. A metric with a
run-to-run spread of ±0.04 behind a min: 0.82 threshold fails builds on
sampling noise alone. The team marks the gate non-blocking, then ignores it,
then deletes it. Comparing single-run point estimates cannot tell you whether a
difference is real.
Silent misses from layer blindness. A RAG system fails in ways the answer hides. The retriever returns the wrong document, and the model writes fluent, plausible, wrong prose over it. Output-level assertions score that as fine, and the regression ships.
EvalFlow's product is not more metrics. It is verdict correctness.
The worked example is a support assistant over a six-document knowledge base,
60 golden examples, evaluated after a re-index gave the refund policy document a
new id. The output below is a real evalflow run on the committed example
(examples/support-assistant) — not a mockup.
$ evalflow run --config evalflow.yaml
support-assistant-regression · 60 examples · retrieval ×1, generation ×5
retrieval
recall_at_k 0.8091 [0.7000, 0.9091] n=55 (5 examples lack expected_documents)
mrr 0.7179 [0.6030, 0.8242] n=55
generation
answer_similarity 0.9165 [0.9068, 0.9254] n=60
exact_match 0.2133 [0.1533, 0.2767] n=60
system
latency_p95 954 ms [922.5, 1006.3] n=60
error_rate 0.0000 [0.0, 0.0] n=60
cost $0.000132 basis bundled-2026.07
gate · paired bootstrap vs runs/baseline · 60 paired cases · 95% CI
✗ recall_at_k −0.1909 CI[−0.30, −0.0909] margin 0.05 regression, sensitivity low
└ smallest confirmable drop 0.1546 vs detection_effect 0.10
✓ answer_similarity +0 CI[0, 0] margin 0.03 pass, sensitivity adequate
└ smallest confirmable drop 0.03 vs detection_effect 0.06
✓ error_rate +0 CI[0, 0] margin 0.01 pass, sensitivity adequate
└ smallest confirmable drop 0.01 vs detection_effect 0.02
advisory thresholds · reported, not blocking in v1
✓ recall_at_k ≥ 0.70 → 0.8091
✓ latency_p95 ≤ 3000ms → 954.4
FAIL · state gated · 300 calls · $0.0397 · exit 1Read what that output actually says, because it is the whole argument for the project:
- Retrieval broke and generation did not. Recall fell 0.19 and the gate
confirmed it — the entire interval lies past the margin.
answer_similaritydid not move at all:+0, interval[0, 0]. An output-only gate would have passed this change without a second glance. The retrieval layer is where the damage is, and only a per-layer gate names it. - The red verdict is honest about its own precision. The regression is
confirmed, but sensitivity is
low: the smallest drop this run could confirm is 0.15, larger than the 0.10 you said you needed to catch. So the drop is real, but its size is a rough estimate — both facts reported, because both are true. - The green on generation is trustworthy, and says why.
answer_similaritypasses withadequatesensitivity: a drop of 0.06 would have been confirmed on this run, so the flat result means "no regression", not "couldn't tell".
About the
+0on generation. The example's offline mock generator writes its answer from the expected output, independent of what was retrieved — so a retrieval regression leaves generation byte-identical. That is deliberate: it is the sharpest possible illustration of failure mode B. A real model leans on its context, so generation would move a little; EvalFlow's job is to make sure the retrieval drop is caught either way. Survivorship handling and error asymmetry (§4.7) don't appear here because the deterministic mock never errors — they are covered in the test suite instead.
Four rules, specified in full in docs/design.md §4.
1. The unit of statistics is the case, never the call. Repeats for one example collapse to a single case value before any cross-case statistic runs. Five repeats over 60 examples is n=60, not n=300. Treating it as 300 would narrow every interval by roughly √5 and manufacture confidence the data does not support.
2. Comparison is paired, and the aggregate is recomputed per resample. Both runs see the same examples, so differences pair up and the test detects far smaller effects. The bootstrap resamples case ids and applies the same set to both sides:
repeat 10 000 times:
C* = sample n case ids with replacement
delta* = agg(current over C*) − agg(baseline over C*)
delta_ci = percentile interval over {delta*}
Recomputing agg per side per iteration is what makes this work identically for
means, p95, and rates — the shortcut of averaging per-case differences is valid
only for means.
3. One test, not two conditions. A regression is confirmed when the whole interval lies beyond a practical margin you configure:
regression iff delta_ci_high < −margin (higher_is_better)
This folds "statistically significant" and "practically large" into a single statement. The burden of proof is on detecting a regression, not on proving its absence — the stricter alternative would fail every build on a noisy dataset and destroy the gate's credibility.
4. Sensitivity is reported separately from the verdict. That permissive rule
has a cost: a real regression can pass on weak data. So EvalFlow also reports
whether it could have caught one — starting from what the rule can actually
confirm. A true drop of size d is confirmed only when d > margin + half_width, so:
minimum_confirmable_regression = margin + (delta_ci_high − delta_ci_low) / 2
That number is printed for every rule, always. Note what it means: a margin of
0.03 does not imply that a 3-point drop gets caught. To judge sensitivity
you also declare detection_effect — the drop size you need to be able to
detect — and the gate is adequate when minimum_confirmable_regression ≤ detection_effect.
detection_effect has no default. Omit it and sensitivity reports
unspecified, never adequate: putting a guessed number behind that word is
exactly the overclaim this rule exists to prevent.
Verdict and sensitivity are independent. The dangerous combination is pass + low: green means not detected, not absent. The tool never reports a green it cannot justify.
A practical consequence worth knowing before you start: gating a binary-ish metric at a 5-point margin needs hundreds of examples, not dozens. At 14 examples this project's own sample dataset could not confirm a 25-point recall drop. EvalFlow tells you that instead of showing you a green check.
schema_version: 3
name: support-assistant-regression
dataset: dataset.jsonl
pipeline:
type: rag
retriever: { type: bm25, index: corpus/, k: 5 }
prompt: prompt.md
provider:
type: openai
model: gpt-4.1-mini
api_key: ${OPENAI_API_KEY}
temperature: 0.0
execution:
repeats: 5 # non-deterministic stages only; BM25 runs once
fail_fast: retrieval
abort_on_error_rate: 0.10
budget: { max_calls: 2000, max_cost_usd: 5.00, on_exceeded: abort }
metrics:
- { name: recall_at_k, params: { k: 5 } }
- { name: mrr, params: { k: 5 } }
- answer_similarity
- exact_match
- latency_p95
- error_rate
- cost
gate:
baseline: runs/baseline/
require_baseline: true
confidence: 0.95
on_low_sensitivity: warn
max_error_asymmetry: 0.02
rules:
- { metric: recall_at_k, margin: 0.05, detection_effect: 0.10 }
- { metric: answer_similarity, margin: 0.03, detection_effect: 0.06 }
- { metric: error_rate, margin: 0.01, detection_effect: 0.02 } # mandatory
thresholds: # advisory in v1
recall_at_k: { min: 0.70 }
latency_p95: { max: 3000 }Three things in there are deliberate and worth explaining:
repeatsapplies per stage, not per run. BM25 over a fixed index returns the same ranking every time, so repeating it buys nothing and costs money. EvalFlow runs deterministic stages once and repeats only what can actually vary. The example runs BM25 60 times (once per example) and the generator 300 times (5 repeats × 60) — the 300 calls in the run above — never paying to repeat a retriever that cannot change its answer.marginis required, with no default;detection_effectis optional but never guessed. How large a drop should block a merge, and how large a drop you need to be able to see, are two different pieces of domain knowledge EvalFlow does not have. Guessing either would undermine every verdict built on top of it.- A gate rule on
error_rateis mandatory. Errored samples are excluded from quality metrics, so a change that fails more often can raise its own scores by leaving an easier surviving set. A config without this rule is rejected.
provider.type: openai needs the optional extra (pip install evalflow[openai]) — the package is imported lazily, only when this provider is
actually built, so the base install never needs it. One deliberate omission:
the run-level seed is never sent to OpenAI's own seed parameter, even
though it exists. Every repeat of an example shares that one seed, and passing
it through would push the API toward repeating itself — quietly narrowing the
very variance repeats exists to measure.
One JSON object per line — streamable, line-diffable, appendable.
{"id": "refund-window", "input": "How long does a customer have to request a refund?", "expected_output": "Customers can request a refund within 30 days.", "expected_documents": ["refund-policy.md"], "tags": ["policy"], "metadata": {"difficulty": "easy"}}expected_documents is what makes retrieval gating possible. Examples without
it — the out-of-scope questions that test refusal, for instance — simply do not
contribute to retrieval metrics and are counted in n_missing, rather than
silently scoring zero. Unknown keys are rejected, so a typo in expcted_output
fails loudly instead of quietly disabling a metric.
A run writes a directory, not a file:
runs/20260728T193359554Z-f6fbb3/
run.json header, metrics with intervals, comparison, decision, budget
cases.jsonl one case per line, dataset order, samples inline
Paired testing needs the baseline's per-case values, so the case data has to
survive; keeping it in a separate file stops run.json from growing with
repeats × cases. Both files are plain JSON, readable and diffable without
EvalFlow installed, and the aggregates are recomputable from the samples.
Off by default — turning it on is a deliberate choice, not a convenience flag:
execution:
cache:
enabled: true
dir: .evalflow/cacheThe reason for the hesitation is specific: a cache is the one execution-layer
feature that can directly corrupt the statistics this project exists to get
right. Every metric here depends on repeats distinct samples per case
forming a real distribution. A cache key built only from (provider, model, prompt) would let repeat 1 quietly reuse repeat 0's cached response —
collapsing every repeat of an example onto a single generation and
manufacturing exactly the false confidence two-stage aggregation and the
paired bootstrap exist to prevent, just one layer further from the statistics
than the bugs that motivated building those in the first place.
So the cache key always includes repeat_index and the run's seed, on top
of the provider's own fingerprint (model, temperature, ...) and the exact
rendered prompt. A provider that can't declare a fingerprint is refused, not
silently left uncached. A price-table change invalidates a cached entry the
same way it invalidates cost gating across a baseline comparison — never a
stale-priced hit. And a cache hit never counts against Budget: it cost
nothing to reproduce, so the artifact's real spend reflects that, while the
cost metric itself is untouched (it already represents one execution's
normalized cost, cached or not).
A cache hit replays the original call's latency, not the microseconds it
took to read the entry off disk. Otherwise a warmed baseline compared against
a cold current run would show a latency "improvement" caused entirely by cache
temperature — and a gate rule on latency_p95 would act on it automatically,
before any human read a disclosure. The lookup cost is reported separately as
a diagnostic, where it cannot reach a metric.
A gate refuses to bless a replayed run by default. Repeat identity keeps cached samples honest within a run, but a fully cache-served run tells you nothing about how the system behaves now — and that's precisely what a verdict claims. So:
gate:
allow_cached_samples: false # the defaultWith the default, a gated run whose samples came from the cache gets
state: replayed, exit 1, and no bootstrap at all — EvalFlow declines to
compute a verdict rather than issue a confident one about a replay. Opt in
with true and gating proceeds, with the replay recorded as a warning in the
artifact. Only the current run's hits matter; a baseline written with caching
on is just a historical snapshot, which is what a baseline is.
Every run that used the cache discloses its hit rate:
cache: 40 hits, 0 misses (100% hit rate) — every generation-derived metric
above reused at least one prior run's response instead of sampling fresh
(latency_p95 still reflects each response's original live call, not the
cache lookup)
cache lookup overhead: 0.016 ms/hit (diagnostic only — excluded from latency_p95)
Present only when caching was enabled — never populated-but-zero when it wasn't, so "off" and "on, and everything missed" stay distinguishable.
This is what a project depending on EvalFlow puts in its own workflow:
name: LLM Evaluation
on: pull_request
jobs:
evaluate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: pip install "evalflow[openai]"
- run: evalflow run --config evalflow.yaml
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}.github/workflows/ci.yml is the different thing:
this repository's own CI, which needs no secrets at all — it runs the test
suite, checks that the mock quickstart passes and that the flagship RAG
fixture's gate still confirms its known regression (exit 1, not a crash), and
builds and installs the wheel into a clean venv to catch packaging bugs a
pip install -e . dev setup can hide.
| exit | meaning |
|---|---|
| 0 | gate passed, or no baseline was resolved and gating was skipped |
| 1 | a regression was confirmed beyond its margin; sensitivity was low with on_low_sensitivity: fail; or the run was replayed from cache without allow_cached_samples |
| 2 | config, dataset, or baseline invalid — nothing ran |
| 3 | aborted — error-rate or budget ceiling exceeded |
Codes 2 and 3 are separate from 1 on purpose: a red build from a real quality regression and a red build from a missing API key demand different reactions.
The first run has no baseline to compare against. That state is explicit —
state: "no_baseline", exit 0, and a report that says plainly no gating
occurred — never an implicit green. Set require_baseline: true to make its
absence an error instead. Writing a baseline is deliberate:
evalflow run --config evalflow.yaml --write-baselineevalflow init # scaffold config and dataset
evalflow run --config evalflow.yaml # execute, gate, write artifact
evalflow run --config evalflow.yaml --write-baseline # establish a baseline
evalflow compare --current runs/<run-id> --baseline runs/baseline
evalflow report --run runs/<run-id> --format json # re-render a saved artifact, offlinereport never re-executes the pipeline and never touches the network — it
reads run.json back and renders it, so it exits with the verdict that run
already reached (0 if it passed, 1 if it didn't). Useful for turning a stored
artifact into a different format after the fact, or reviewing an old run
without spending another API call.
Not on PyPI yet — install from a clone. Requires Python 3.11+; CI runs 3.11, 3.12, and 3.14.
pip install -e . # core + mock provider — no extras needed
pip install -e ".[openai]" # + the real OpenAI adapter
pip install -e ".[openai,dev]" # + pytest, for running the test suitepytest63 tests run offline, against the mock provider and an injected fake OpenAI
client — no network, no key. Installing the openai extra widens what that
offline suite covers (it exercises the real SDK's client-construction path
via a monkeypatched class) but still makes no network call on its own.
Two more tests make a real, billed call to the OpenAI API and are skipped
unless OPENAI_API_KEY is set — run them on purpose, not by accident:
OPENAI_API_KEY=sk-... pytest tests/test_openai_live.py -m live -vThey use the cheapest model (gpt-4.1-nano) and a couple of one-word prompts;
cost is a fraction of a cent.
v1: RAG pipeline with BM25; mock provider plus one real provider;
recall_at_k, mrr, token-overlap answer_similarity, exact_match, latency,
tokens, cost, error_rate; per-stage repeats; paired bootstrap; margin gate;
sensitivity reporting; survivorship accounting; an off-by-default response
cache; JSON and Markdown reports; GitHub Actions example.
Deliberately out of v1 — named so they are not mistaken for oversights:
LLM-judge metrics, sequential testing, flake quarantine, evalflow power,
multiple-comparison control, rerankers, vector retrievers, extra providers,
blocking absolute thresholds, web UI, REST API.
Not in scope at all: red-teaming and security scanning (promptfoo does this well), prompt playgrounds, production observability, public model leaderboards. EvalFlow measures your system on your data.
- docs/positioning.md — the product bet, and what it rules in and out
- docs/design.md — domain model, statistics, formats, exit codes
- examples/support-assistant — the worked example above, with artifacts
MIT.