Repository navigation
Pull request backtests in CI
On a pull request that changes alert rules, the backtest can comment its report on the pull request, and edit that comment on each push: what fired before the change and what fires after, rule by rule, over the period you name. recipes/backtest has a script for each code host, GitHub, GitLab, Bitbucket, Azure DevOps and Gitea (or Forgejo), one library under them; and, for GitHub Actions, an action:
on:
pull_request:
paths: ['alerts/**']
permissions:
contents: read
pull-requests: write
jobs:
backtest:
if: github.event.pull_request.head.repo.full_name == github.repository # never a fork's
runs-on: ubuntu-latest # one that can reach your Prometheus
steps:
- uses: actions/checkout@v4
with: {fetch-depth: 2, persist-credentials: false}
# Pinned by the commit of the release's tag, which a tag moved can't
# change; the action installs that release of lapilli.
- uses: lapilli-project/lapilli/recipes/backtest@COMMIT-OF-THE-vX.Y.Z-TAG # vX.Y.Z
with:
paths: alerts
prometheus-url: ${{ vars.PROMETHEUS_URL }}
since: 14dEach installs a pinned release, checked against its SHA256SUMS (the
action, the release it is at, which Dependabot keeps up to date with the
uses: line); reads the
rules of the pull request and of the commit it goes into, as files
(paths) or as a command renders them in each (render: helm template monitoring charts/monitoring), and the incidents from the commit it goes
into, never the pull request's (incidents), and the times a PromQL
expression says users were hurt (impact, match: service); runs lapilli verdict --compare --changed-only --markdown; and posts one comment, found again by
a hidden first line. When nothing that decides when a rule fires changed
and there is no comment yet, it posts none. When something fails, the comment says
what, and the job succeeds unless it is asked to fail. The recipes' README
has each CI's job, and what to keep from pull requests from forks: their
rules become queries sent with your credentials, and a render command runs
what they hold.
--markdown writes the report for a pull request or a merge request, at
most 64,000 bytes, under the 65,536 characters GitHub takes, cut short
with a note when longer:
[//]: # (lapilli-compare {"nothing_changed":false,"changed":1,"added":1,"removed":0,"same":42,"alike":0,"unreplayed":0,"recording_changed":0,"firings_before":148,"firings_after":42,"incidents_given":true,"incidents":3,"incidents_changed":1,"no_record":false})
### Lapilli backtest, 2026-10-26 00:00 → 2026-11-02 00:00 Asia/Seoul (+09:00)
Rules compared: **1 changed, 1 added, 0 taken away**, 42 the same.
| | Before | After |
|---|--:|--:|
| Firings | 148 | 42 |
| Severity `critical` | 125 | 16 |
| Severity `warning` | 23 | 26 |
| Incidents with no alert, of 3 | 1 | 0 |
#### Rules that differ
| Rule | Severity | Fired | ○ Caught | ● Nothing | Precision | Refired |
| --- | --- | --: | --: | --: | --: | --: |
| `ExportJob5xx` | `critical` | 118 → 9 | 0 → 0 | 118 → 9 | 0% → 0% | 41 → 2 |
| `DiskFillingUp` | – → `warning` | – → 3 | – → 1 | – → 2 | – → 33% | – → 0 |
#### Incidents caught differently
| Incident | Start | Before | After |
| --- | --- | --- | --- |
| `INC-43` `Batch stuck` | 2026-10-31 02:40 | ✗ nothing fired | `DiskFillingUp` +4m |
When rules that differ couldn't be replayed, a line under the totals says
how many, as unreplayed counts them, since the numbers leave them out;
they are listed, with why, under the table. Those that differ but fire as
they did are folded in <details>.
A recording rule that differs is a change too, listed apart: a replay reads
what a recording rule recorded, from the history, so the alerts that read
one are replayed against what it recorded, not what the change would
record, which the report says. Recording rules are read wherever
Prometheus's groups are (files, a PrometheusRule, a ConfigMap, Google
Cloud's Rules, an Amazon rule groups namespace, a Mimir, Cortex or Loki
ruler's file), from an Azure rule group, alone or in an ARM template, and
from a Grafana provisioning file, where a recording rule (one with
record) is no alert rule. A cloud's own alarms record nothing.
A group's labels and limit are part of each of its rules, as
Prometheus 3 runs them: its alerts carry the group's labels under the
rule's own, and a change to either is a change to the rule. A limit isn't
replayed: past it, Prometheus drops all of a rule's alerts, which the
report says too. A rule's name, and in lapilli serve its versions,
history and proposals, are as they were before Lapilli read them: a rule
told apart from another of its name is told apart by its group and its own
labels. Not every server that takes Prometheus's groups applies these:
Prometheus 2 refuses a group's labels; Google Cloud's Rules,
ClusterRules and GlobalRules have neither field; and a Mimir, Cortex or
Amazon ruler may keep them without applying them, depending on its
version. Where they aren't applied, the replay's alerts carry labels the
real ones don't.
Rule
names, labels, incidents and the reasons a rule couldn't be replayed come
from whoever wrote the rules, and a pull request's author may be anyone:
each is written as a code span, in which no host reads Markdown, HTML, a
mention (@user, @org/team), a reference (#123) or a link, a |
escaped so it can't end a table's cell, a backtick in it making the span's
fence longer, and control and format characters (bidirectional overrides,
zero-width spaces), line separators and spaces other than the space shown
as \u{...}. A query that failed as the history couldn't be read (as
--fail-unreplayed counts it: anything but the history's answer to the
query as written) says only that in the Markdown, its whole reason going to
stderr, the job's log; what is left, the history's own words on a query it
refused, has where the history and the rules are read from taken out: the
--prometheus, --loki, --grafana and --ruler URLs, the user names
in them and their hosts, each as a whole word, any IP address and port,
the tenants and a --gcp-project.
What a tool reads is stable:
-
The report of
--jsonand/api/v1/report, the columns of--csv, and the names and labels of--metricsand/metrics: those docs/numbers.md lists keep their names and meaning; a minor release that must change one says so in the changelog and writes the old beside the new for a release first. -
The first line of
--markdownis a link reference definition, which no host shows, holding the summary as JSON;--jsongives the same object as"summary", beside the rest of the report, whose fields are as they were. Every field is always there:Field nothing_changednothing that decides when a rule fires differs, a recording rule included: nothing was replayed changed,added,removed,samerules in both that differ, only in --rules, only in--compare, alike in bothalikerules that differ but, replayed on both sides, fire as often and catch as much as they did unreplayedrules that differ that couldn't be replayed, on one side or both, for any reason: those of --json's"unreplayed"list with a side, each counted oncerecording_changedrecording rules that differ, listed as "recording": what the alerts that read them see, which no replay showsfirings_before,firings_afterfirings on each side; of the rules that differ alone with --changed-onlyincidents_givenincidents were given, --incidentsor--impact; when false, nothing was judged: no firing caught anything, andincidentsis 0. The firings were judged only when it is true andno_recordfalseincidents,incidents_changedincidents in the period, and those caught differently no_recordincidents were given, and none was on in the period or the two weeks before it (nor --impactread): nothing was judged, as without themexpectedthe rules that differ were held to expectations, --expectexpectations_brokenexpectations the change broke: met by the rules in place (or of a rule it adds), broken by those proposed expectations_already_brokenexpectations broken on both sides, which aren't the change's unexpectedrules the change adds that no expectation holds -
When nothing changed,
--markdownwrites that line and exactly:### Lapilli backtest: nothing changed Nothing that decides when the 42 rules fire has changed: nothing to replay. -
The exit status is
lapilli verdict's, as it was: 0 when the report is written, whether or not anything changed; 1 when it failed, or a rule's replay didn't finish within--timeout(the report is still written), or, with--fail-unreplayed, the history couldn't be read for one or its labels on its series were too many to replay, or, with--fail-expectations, the change broke an expectation (and with--expect-required, it added a rule none holds); 2 when the flags are wrong. Nothing changing has no exit status of its own: a CI step takes anything but 0 as a failure, and a pull request that changes no rule isn't one. Readnothing_changed.
The header names the time zone of its times with each offset it took in
the period, Asia/Seoul (+09:00), Europe/Berlin (+01:00 → +02:00), or
UTC by any of its names: --tz gives one by its IANA name, and without
it, the machine's own is named as TZ or /etc/localtime says, through
every link. A zone file copied rather than linked has no name: its
abbreviations are said, CEST (+02:00) → CET (+01:00).
A CI job can send a pull request's rules to lapilli serve instead of
running Lapilli itself: serve replays them with its own access to the
history, at its share of the rate, judges them by its record, and gives
back the same report, summary line and all, which the job posts as it
would its own, the lines Lapilli writes and no other (serve.md; the design,
design/backtests.md). The job holds no credential
for the metrics, only a token that says which repository it runs for: the
CI's own ID token where it has one (GitHub Actions, GitLab CI, Forgejo
Actions), a plain token elsewhere. In the recipes, it is LAPILLI_SERVE,
and on GitHub Actions the action's serve-url and audience, with
id-token: write:
permissions:
contents: read
pull-requests: write
id-token: write
jobs:
backtest:
if: github.event.pull_request.head.repo.full_name == github.repository # skips a fork's run; GitHub gives one no ID token
runs-on: ubuntu-latest # needs no access to the metrics
steps:
- uses: actions/checkout@v4
with: {fetch-depth: 2, persist-credentials: false}
- uses: lapilli-project/lapilli/recipes/backtest@COMMIT-OF-THE-vX.Y.Z-TAG # vX.Y.Z
with:
paths: alerts
serve-url: https://lapilli-ci.example.com
audience: https://lapilli-ci.example.com
since: 14dThe if: saves a fork's run the job; it keeps no one out, as the pull
request can change it. What keeps a fork out is GitHub giving a fork's
pull_request run no ID token, unless, on a private repository, "Send
write tokens to workflows from pull requests" is on: keep that setting off
(serve.md). The trust entry's pins
don't: a fork's run in the repository carries the repository's own
repository_id.
The rules are read from git, and render run, in the job as before;
expect is read from both commits and held by serve, and the job's gate
(fail-on-expectations, expect-required) reads the summary line as it
reads Lapilli's. Serve decides where the history is and what judges the
firings, so the settings for those (prometheus-url, incidents,
impact, match, --lead, --from, ...) are refused with it: of the
arguments, --since (whole days, 1 to 14), --tz and --rules-select
are sent. The comment's heading says it was judged by lapilli serve's
record, and reads differently where that record does: catches judged
against the incidents and impact windows serve keeps, firings by its weeks
and settings. The recipes' README has each CI's token, what is sent, and
what is refused (Through lapilli
serve).
This page is copied from docs/pr-backtests.md, where it is changed: a pull request there changes it here.
Start here
Judging a period
- How it judges
- Use
lapilli verdict - The numbers
- A cloud's own alarms
- Rules on logs: Loki
- Is the replay right?
- What an alert says
Before a change
Incidents
Reports
Every hour: lapilli serve
Reference
And
Also at lapilli.dev/docs.