Repository navigation
Before a change is merged
--compare sets the rules proposed beside the rules in place: both are
replayed over the same period and judged against the same incidents, and
the report shows what the change does. Each rule that differs gets its
stones and severity before → after, firings are counted by severity, and
each incident caught differently is listed with both catches. Rules alike
in both are replayed once, for both; --changed-only doesn't replay them
at all, which is much faster, but then an incident shows only the changed
rules' catches.
$ lapilli verdict --rules proposed/ --compare alert-rules.yaml --prometheus http://prometheus:9090 \
--incidents incidents.yaml
Lapilli 2026-10-26 00:00 → 2026-11-02 00:00 Asia/Seoul (+09:00)
Rules compared: 1 changed, 1 added, 0 taken away, 42 the same
Firings 148 → 42
By severity: critical 125 → 16, warning 23 → 26
Of 3 incidents: with no alert 1 → 0
RULE SEVERITY FIRED ○ CAUGHT ● NOTHING PRECISION REFIRED
ExportJob5xx critical 118 → 9 0 → 0 118 → 9 0% → 0% 41 → 2
DiskFillingUp – → warning – → 3 – → 1 – → 2 – → 33% – → 0
INCIDENT START
INC-43 Batch stuck 2026-10-31 02:40
before ✗ nothing fired
after DiskFillingUp +4m
Without --incidents or --impact, --compare still runs: the firings
are counted, not judged, and the report leaves out what only incidents
could say (caught, nothing, precision, the incidents caught differently),
keeping how often each rule fires and re-fires. Rules left alone are
replayed too, once, so an incident one of them catches is caught on both
sides: a longer --since judges against more incidents, and takes longer.
In CI, run it on each pull request that changes the rules
(Pull request backtests in CI).
--expect FILE holds the rules that differ to what is expected of them,
written beside them (docs/design/expectations.md):
how many times each may fire in a week, how many of those while people
slept (--sleep-hours, 22:00–08:00 by default), and the incidents it must
keep catching, within how long. A rule is named, or a group's rules:
expectations:
- rule: CheckoutErrors
budget:
week: 10 # firings in each whole week of the period
asleep: 2 # of them, while people slept
catches:
- incident: INC-43
within: 5m
- group: payments
budget: {week: 20}A budget is held to each whole week of the period, counted back from its end, and the week that fired most is shown; a period shorter than a week is scaled to one, and says so. A catch is judged as the verdict judges incidents: an incident not in the period, or no incident given, is not known, never met or broken; a rule the change takes away catches nothing. The report gets a section, each expectation before → after: met, broken by this change (met before, or a rule the change adds), broken before too (which the change isn't held to), or not known.
Expectations: 1 broken by this change, 0 broken before too, 2 met
Budgets are held to each of the period's 2 whole weeks, counted back from its end: the most in one is shown.
RULE HELD TO BEFORE AFTER
CheckoutErrors at most 10 a week 3 41 ✗ broken by this change
CheckoutErrors at most 2 a week while people slept 0 1 met
CheckoutErrors catches INC-43, within 5m +2m +2m met
Entries for the same rule or group are made one when read, and the same
budget or incident given twice is refused. A name several rules share is
told apart in a run (CheckoutErrors [severity=page]): rule: CheckoutErrors holds each of them, and the name told apart one. A rule a
change moves out of a group that held it is said. --expect-before FILE gives the expectations as they were,
for the rules in place: a budget the change raises is held to its old value
before it, and the report says it was raised; one it adds is held on both
sides and said new; one it takes away, whole or a field of it, is still
shown, held to what it was, and fails nothing, as a budget raised doesn't:
loosening is for the reviewer to see. --fail-expectations exits 1 when the
change breaks one; --expect-required when it adds a rule no expectation
holds. Nothing here needs incidents: a budget holds a rule with none
written down. The file is read strictly, a key it doesn't know an error,
1 MiB, 2,000 expectations and 2,000 catches at most, and before anything is replayed. The backtest recipes read it from both
commits (EXPECT, Pull request backtests in CI). The rules that differ are held to 100,000
expectations at most, each rule to each budget and catch that holds it (a
group's catch holds every rule of the group the change touches); past that,
the run fails and says so.
This page is copied from docs/compare.md, where it is changed: a pull request there changes it here.
Start here
Judging a period
- How it judges
- Use
lapilli verdict - The numbers
- A cloud's own alarms
- Rules on logs: Loki
- Is the replay right?
- What an alert says
Before a change
Incidents
Reports
Every hour: lapilli serve
Reference
And
Also at lapilli.dev/docs.