Repository navigation
Incidents kept elsewhere
The thinner the incident log, the worse every verdict, so copy it from where
it is kept. recipes/incidents has a small script for each
of PagerDuty, incident.io, Jira, GitHub issues, Grafana IRM, Rootly,
FireHydrant and Compass's alerts (for Opsgenie's users, whose incidents move to
Jira Service Management, which the Jira one reads). Each writes what
--incidents takes, so it works with both commands:
PAGERDUTY_TOKEN_FILE=pd-token SINCE=2026-10-01 recipes/incidents/pagerduty.sh > incidents.tmp &&
mv incidents.tmp incidents.yaml
lapilli verdict --rules alert-rules.yaml --prometheus http://prometheus:9090 --incidents incidents.yamlMove the file into place: an incidents file with nothing in it is refused, so
that a script that failed doesn't read as a week with no incidents. Write
incidents: [] for that.
lapilli serve can also take them as they come. With --incidents-listen ADDR
and --incidents-token-file FILE, it opens a port that takes incidents from
whoever holds the token, and does nothing else. A job can run this every hour:
an incident written again as it was changes nothing, tells no one, and judges
nothing again. Unlike a file of the last month, it forgets nothing older.
recipes/incidents/pagerduty.sh > incidents.json &&
LAPILLI_TOKEN_FILE=token recipes/incidents/put.sh http://lapilli:8082 < incidents.jsonThe port is its own so that a NetworkPolicy can let the job reach it alone. On the UI's port a pod could name itself anyone, in the headers the sign-in proxy sets; here those headers mean nothing. What the job copies in is the tool's: the UI can't change it, and the token can't change what people wrote. It isn't counted as anyone's use of Lapilli, and isn't told to the notice webhook, since the tool has told people already. The port takes one write at a time, 20 a second, and the ledger keeps 20,000 copied incidents at most.
This page is copied from docs/incidents.md, where it is changed: a pull request there changes it here.
Start here
Judging a period
- How it judges
- Use
lapilli verdict - The numbers
- A cloud's own alarms
- Rules on logs: Loki
- Is the replay right?
- What an alert says
Before a change
Incidents
Reports
Every hour: lapilli serve
Reference
And
Also at lapilli.dev/docs.