Repository navigation
Getting started
From installing Lapilli to a first report on your own alerts, with the
rules you have now and a Prometheus-compatible API that holds their
history. lapilli verdict only reads the history and files, and sends the
report nowhere but where you say.
go install lapilli.dev/cmd/lapilli@latestwith Go 1.26.11 or later, as go.mod says; or take a binary from the
releases (macOS and
Linux, amd64 and arm64), or the image ghcr.io/lapilli-project/lapilli,
whose entrypoint is lapilli.
Checking a release says how to check
what you took, offline too.
Each firing is judged against what went wrong, so Lapilli needs a record of it. The smallest is a YAML file:
incidents:
- id: INC-100
title: Login slow
start: 2026-11-03T08:10:00Z
end: 2026-11-03T08:50:00Z
labels: {service: login}
- id: INC-101
title: Checkout errors
start: 2026-11-11T10:02:00Z
end: 2026-11-11T10:45:00Z
labels: {service: checkout}
- id: INC-102
title: Search timeouts
start: 2026-11-13T15:30:00Z
labels: {service: [search, catalog]}
- id: INC-103
title: Orders stuck
start: 2026-11-14T03:05:00Z
end: 2026-11-14T03:30:00Z
labels: {service: orders}-
idandstartare required: an id each once in the file, and an RFC 3339 time with its offset (Z,+09:00). -
end,titleandlabelsare optional. Without anend, an incident is taken to last an hour; one can't be beforestart.labelssay where it went wrong, a value or a list of values a label: with--match service, a firing with aservicelabel catches only an incident that lists its value or has noservicelabel; a firing or an incident without the label isn't held to it.
Write down the incidents of the two weeks before the period too: a firing
that caught nothing is judged only when an incident was on at some time in
the two weeks before it, or --impact was read over its time; else it is
counted, not judged (No record). INC-100 is there
for that: it isn't listed, being before the week judged below, but every
firing of the week is judged.
A file with no incidents key, an empty one among them, is refused, so a
job that failed doesn't read as a week with nothing wrong: write
incidents: [] for none. Or have incidents another way:
- Incidents kept elsewhere: scripts that write this file from PagerDuty, incident.io, Jira, GitHub issues and other tools.
-
Incidents nobody wrote down:
--impact PROMQL, windows of a signal of user impact taken as incidents, beside the file or alone.
lapilli verdict --rules alert-rules/ --prometheus http://prometheus:9090 \
--incidents incidents.yaml --match service--rules replays the rules against the history at --prometheus:
Prometheus rule files, PrometheusRule resources, Grafana alerting
provisioning files, ConfigMaps holding them, or a directory of them. The
period is the last 7 days (--since 7d), or --from to --to in RFC
3339, its times shown in the machine's zone or in --tz. --match service
keeps an alert about one service from catching an incident in another
only because they came at the same time; without it, any firing in time
catches.
In place of --rules alert-rules/:
-
--loaded-rules: the rules the API at--prometheushas loaded. -
--grafana https://grafana.example.com --grafana-token-file grafana-token: the rules a Grafana holds, read with a Viewer's service account token,--prometheusstill given. -
--alerts: what Prometheus fired, from itsALERTSseries, with no replay; so no longerforis suggested and no near misses are looked for.
For a cloud's managed Prometheus, --auth aws, azure or google sends
the credentials the machine or pod already has, and --bearer-token-file FILE a token: Use lapilli verdict has each cloud's URL and
the permission it needs.
Over a made-up week, given with --from, --to and --tz UTC, with the
incidents above, three rules (CheckoutErrors, NodeLoadHigh,
QueueBacklog) and a stand-in answering for Prometheus, the command above
writes this, three of its parts left out:
Lapilli 2026-11-09 00:00 → 2026-11-16 00:00 UTC
14 firings, 3 incidents written down
RULE SEVERITY FIRED ○ CAUGHT ● NOTHING PRECISION REFIRED
NodeLoadHigh warning 13 1 12 8% 0
CheckoutErrors critical 1 1 0 100% 0
Noisy rules that a longer "for" would quiet, still catching every incident they caught:
RULE FOR FIRED INCIDENTS CAUGHT LATER BY
NodeLoadHigh 1m → 10m 13 → 1 1 up to 9m
INCIDENT START LASTED CAUGHT BY
INC-101 Checkout errors 2026-11-11 10:02 43m CheckoutErrors +3m
INC-102 Search timeouts 2026-11-13 15:30 – NodeLoadHigh -9m (noisy)
INC-103 Orders stuck 2026-11-14 03:05 25m ✗ nothing fired
✗ 1 of 3 incidents had no alert.
! 1 of 3 incidents were caught only by noisy rules, which fire whatever happens
(10 or more firings, under 20% of them catching anything).
Rules that nearly caught them: their condition held, but not for their "for", or fired too late; or their value came within a fifth of their threshold.
INCIDENT RULE HELD CHANGE WOULD FIRE FIRED ● NOTHING
INC-103 QueueBacklog 7m from -5m for 10m → 6m +1m 0 → 1 0 → 0
From the top:
- The period and how many firings and incidents it holds; a line under it says how many firings had no record to judge them by, when some had.
-
Each rule that fired: its firings, those that caught an incident (○,
white stones), those that caught nothing (●, black stones), its
precision (the share of its judged firings that caught something), and
those that came within
--refire(1h) of the same alert resolving. -
Noisy rules, 10 firings or more with under 20% catching anything,
replayed with a longer
for: the shortest that keeps every incident each caught, firing at most half as often. -
The incidents: how long each lasted (
–with no end written down), what caught it and how long after its start (+3m) or before (-9m);(noisy)marks a noisy rule's catch,✗ nothing fireda miss. -
Rules that nearly caught them: a rule whose condition held but not
for its
for, or whose value came within a fifth of its threshold, with the change that would have caught the incident and how often it would then have fired. With aforof 6m,QueueBacklogwould have fired a minute into INC-103, once in the week.
Left out above: the firings that caught nothing by when they came in
--tz (while people slept, off hours, at work); what each rule adds (the
incidents each caught, first and alone, and the rules that are spare); and
the rules that never fired or couldn't be replayed, with why.
How it judges says what the report rests on;
The numbers defines each number and how to count it again
from --json, the whole report, every firing included.
Before a rule is changed, --compare replays the rules proposed beside the
rules in place, over the same period and against the same incidents
(Before a change is merged):
lapilli verdict --rules proposed/ --compare alert-rules.yaml --prometheus http://prometheus:9090 \
--incidents incidents.yamlThe Helm chart runs lapilli verdict every Monday at 09:00 over the week
before, and sends the report to Slack and the metrics to a Pushgateway
(Every week).
lapilli serve keeps judging, every hour, into a ledger with a web UI
(Keep judging). lapilli preflight takes its flags and says
what it will meet: the rules, the history, and the requests and memory its
runs will take. It writes and replays nothing, and exits 0 with no problem
(Before it is deployed):
lapilli preflight --rules alert-rules/ --prometheus http://prometheus:9090 \
--incidents incidents.yaml --match servicelapilli serve with the same flags replays the last 28 days on its first
start (--backfill), keeps the ledger in lapilli-ledger, and listens on
127.0.0.1:8080, where everyone who reaches it is an admin. In Kubernetes,
the chart needs no more than this:
helm install lapilli oci://ghcr.io/lapilli-project/charts/lapilli -n monitoring \
--set mode=serve --set prometheus.url=http://prometheus:9090 \
--set 'rules.configMaps={alert-rules}' --set 'match={service}'
kubectl -n monitoring port-forward svc/lapilli 8080rules.configMaps names the ConfigMap of rules, in the release's
namespace; incidents takes the same list as the file, or people write
them down in the UI. With no roles, everyone who reaches the pod may only
look (Who people are).
This page is copied from docs/getting-started.md, where it is changed: a pull request there changes it here.
Start here
Judging a period
- How it judges
- Use
lapilli verdict - The numbers
- A cloud's own alarms
- Rules on logs: Loki
- Is the replay right?
- What an alert says
Before a change
Incidents
Reports
Every hour: lapilli serve
Reference
And
Also at lapilli.dev/docs.