Skip to content

Is the replay right

github-actions[bot] edited this page Nov 18, 2026 · 1 revision

Every verdict rests on the replay, so check it on your own rules. With --check-replay, lapilli verdict replays the rules and sets them beside the series that say what really fired, instead of judging. It needs no incidents.

$ lapilli verdict --rules alert-rules.yaml --prometheus http://prometheus:9090 --check-replay
Replay beside ALERTS{alertstate="firing"}  2026-10-26 00:00 → 2026-11-02 00:00 Asia/Seoul (+09:00)
139 firings alike, 2 only in the replay, 7 only in what really fired: 94% alike.
21 rules fired alike, 2 differ, 38 fired neither way.

RULE          REPLAYED  REAL  ALIKE  ONLY REPLAYED  ONLY REAL  OFF BY
ExportJob5xx  118       122   116    2              6          30s
QueueDepth    9         10    9      0              1          1m0s

Really fired, but no rule given has the name: Watchdog.

Two firings are alike when they are of the same alert, agree on the labels both have, and start within two evaluations of each other, since Prometheus places a group anywhere in its interval. Labels a store adds to the series are passed over, and so is a rule's label with a template in it. A firing with no match too near the end of the period to tell isn't counted. A group's interval is taken from the rules; a rule file that leaves it to the global evaluation_interval doesn't say, so give that as --step if it isn't 1m.

What really fired comes from a series, or from Grafana's own history:

Rules evaluated by What really fired
Prometheus ALERTS{alertstate="firing"} documented
Amazon Managed Service for Prometheus ALERTS{alertstate="firing"} documented
Grafana, with --grafana or --check-grafana Grafana's own alert state history, read through its API kept in annotations, its default, from Grafana 12.4 on, or in Loki. Kept in Prometheus too, the series below while it holds anything
Grafana GRAFANA_ALERTS{alertstate="firing",grafana_alertstate!="error"} once state history is written to Prometheus. Its error alerts are left out: a query that failed can't be replayed
Thanos, Mimir, Azure Monitor, Google Cloud ask for ALERTS and see their docs don't say; --check-series names another series

Grafana keeps its alert state history in annotations unless told otherwise, and writes GRAFANA_ALERTS only when told to, so most Grafanas have no series to check against. Given the Grafana, Lapilli reads the history it keeps instead, with the token --grafana-token-file holds, a Viewer's: --grafana for the rules it reads from there, and --check-grafana URL for Grafana's rules read from files.

$ lapilli verdict --rules grafana-alerting/ --prometheus http://prometheus:9090 \
    --check-replay --check-grafana https://grafana.example.com --grafana-token-file token
Replay beside Grafana's alert state history (annotations)  2026-11-09 00:00 → 2026-11-16 00:00 UTC

The history is read from a day before the period, as Grafana keeps changes of state, not states: an alert on since before the period is seen as on at its start, and one that fired through all of it, changing state in none of it, isn't seen. Its error alerts are left out and said, as with the series. Annotations keep an alert's labels as Grafana writes them as tags, a colon an underscore (instance:10.0.0.1_9100), spaces trimmed, a label past 100 characters or past [annotations] tags_length left out; the replay's are set beside them so. Grafana's rules aren't checked, and the others are, when the history can't be read, holds no change from before the period (it may keep none that far back: shorten it), or tells a rule's alerts apart by nothing it keeps; each says why. On Grafana 12.4 and 13.3, every firing read from the history was alike with GRAFANA_ALERTS, starting and ending at the same evaluation (docs/design/grafanahistory.md). --check-series reads series instead.

When the series hold nothing for the period, or no rule could be replayed, it says so and exits 1, since nothing was checked. If a rule replays differently from how it really fired, please report it: it is the most useful report there is.

lapilli serve does this itself, as it goes (docs/design/fidelity.md). Each day the runs won't replay again is set beside the same series, each firing counted on the day it started, and kept. A rule's page says, over twelve weeks, how many were alike, how many only in the replay and how many only in what really fired, and lists the last of those; the rules list and the dashboard mark a rule whose replay differed over the last week (five firings either way, fewer than nine in ten alike, unless --check-differs N/RATIO says otherwise), and replay.differs tells the webhook and Slack once a week while it does. The Run history page says how far it has got, which series held nothing (a store may not keep ALERTS; --check-series names another), whether Grafana's own history stands in for GRAFANA_ALERTS, a day it held nothing from before it (Grafana may keep none that far back), more than is read at once, or what couldn't be used, when Grafana's rules aren't checked that day and the check goes on, and what really fired with no rule given by that name. /metrics has lapilli_rule_replay_check{rule, way} over seven days, MCP's rules and rule carry it, and the weekly digest has a line of it. No stone changes. A cloud's own PromQL alarms (CloudWatch, Google Cloud, Azure) keep no such series: read with --alarms SOURCE=URL, their history copied in under SOURCE stands in, by each firing's alarm_id, a day checked once the copies reach past it (waiting two days at most); a CloudWatch alarm's replay is first put together as CloudWatch keeps the alarm, as a whole. lapilli verdict --check-replay takes the same from a recipe's output with --check-firings FILE. The replay checked is the one judged, with --late alike: an alarm replayed without it reads its metrics as filled in since, where the alarm saw them as they arrived, and may differ for that. Rules that share an alert name are counted together, as what really fired doesn't tell them apart. The series are trusted as they are: a scraped target that exposes its own ALERTS would be counted. It is on whenever there are rules to replay; --check-replay=false turns it off, and so does the chart's serve.checkReplay.enabled.


This page is copied from docs/check-replay.md, where it is changed: a pull request there changes it here.

Clone this wiki locally