Repository navigation
What it doesn't do yet
github-actions[bot] edited this page Nov 18, 2026
·
1 revision
- Queries to data sources that are neither PromQL nor LogQL (SQL, Elasticsearch, InfluxDB, a cloud's own logs) can't be replayed; such rules are listed with the reason. A Loki ruler's group runs a fraction of a second past a second, which the replay evaluates on: a log line written in that fraction is counted an evaluation later.
- Query errors aren't replayed: history can't say when a query failed.
--check-replaysays how many of Grafana's error alerts it left out of what really fired. - A query that has returned nothing for over a day, or never in the history there is, is taken to have no history, not no data: such a Grafana rule is replayed as quiet, where Grafana may have sent NoData.
- A group without an
intervalis evaluated every--step(1m): the globalevaluation_intervalisn't in rule files, nor is the globalrule_query_offset. An interval that isn't a whole number of steps is evaluated every step. Prometheus places a group within its interval by a hash of the path it loaded the file from, which a file doesn't say: read from a file, a firing can start up to an interval sooner or later than it did.--loaded-rulesknows where. - Grafana reduces a query's points as far apart as its
intervalMs(1s by default) or more over a long window; lapilli asks for them so forcount,count_non_nullandsum, and for the rest at--scrape-interval, or else at the shortest interval the series they read are scraped at, as/api/v1/status/configand/api/v1/targetssay, or, where neither is served (Mimir, Thanos, a managed Prometheus), at Grafana's 1s, which costs the server fifteen times and more the work: give--scrape-intervalthere. At the scrape interval, the step is cut to one that divides the rule's interval and the window too, so a sample is seen at the first evaluation after it, as Grafana sees it (a rule every 10s on series scraped every 15s is asked every 5s). - A Grafana rule's query on another data source than its condition's (a Loki query whose values an annotation shows) isn't replayed: when it returns nothing, Grafana takes the rule as having no data, which the replay can't see, and the rule is marked "replayed in part".
- Replays see the history as it is now.
--late DURreplays as if metrics arrived DUR after their time, but a delay that varies can't be read from history: a rule that resolved and fired again while the latest points were missing is replayed as one firing. Rules on a cloud's own monitoring are marked "replayed in part" until--latecovers them. - Rules alike in name, group and labels are counted together, and so are
firings read with
--alertsor--firings, which carry no group. -
lapilli serveruns as one replica on a ledger file, or as several on one Postgres database, one of them leading (docs/design/replicas.md). Who is who comes from its sign-in (an OpenID Connect provider, or a sign-in in front), and roles from the flags and the ones admins give; it doesn't sign people out at the provider, or refresh a session past--session-max-age. A version of the rules is in effect from when it saw it, up to--watch(1m) later, and for a mounted ConfigMap a minute or so more; the rules before its first start are taken to be those it found then, unless--rules-historysays how they were. It gives a firing its stones against every incident it has, so a firing just before a week starts can catch an incident in that week, wherelapilli verdictover that week alone sees only the week's firings. - It counts firings, and the notifications they became only where the alert
manager sends them to
--notify-listen: Alertmanager's routing, grouping, inhibition and silences, and a Grafana notification policy, aren't read from their configuration. A firing that paged no one can't say why: silenced, inhibited and routed nowhere look the same from the receiving end. Notifications to PagerDuty, Slack and the like are seen only where the alert manager also sends them to Lapilli, or, from Alertmanager 0.33.0 on, its event recorder tells of them; the recorder's mutes, which say why a firing paged no one, aren't kept yet. - Alarms kept in a cloud's own monitoring are replayed only where they are
written in PromQL (CloudWatch PromQL alarms, Google Cloud policies with a
PromQL condition, Azure's Prometheus rule groups). The rest are judged from
their history, with
--firingsor copied intolapilli serve, and have no change reviews, longerfors or near misses. - A CloudWatch PromQL alarm and a Google Cloud PromQL policy keep no series
of what they fired: their replay is checked against their history only
where it is copied in (or given with
--check-firings), and CloudWatch's only as the alarm as a whole. - Changes are carried out on four paths: a pull request where the rule is
kept in git, Grafana's API, a managed Prometheus ruler's API (Mimir,
Cortex, Amazon's, Azure's), and a cloud's own alarms' API (CloudWatch,
Azure Monitor, Google Cloud Monitoring, OCI Monitoring), none of the
clouds' run against an account yet. An alarm that isn't PromQL gets no
foror threshold from Lapilli (nothing could backtest it), and is never deleted from here. Which tool made an alarm isn't asked of a cloud's deployment records: its tags and--changes-indecide. A Grafana rule's threshold is changed by a person, in Grafana. Google Cloud'sRulesresources are Kubernetes resources, changed by the git path. Nothing on a Mimir's or a Cortex's group says which tool keeps it, and no ruler has a condition on a write: a write landing between Lapilli's read and its own is found when the change is read back, and fails. - Pull request backtests are checked end to end against a real Gitea; the recipes for GitHub, GitLab, Bitbucket and Azure DevOps against stand-ins for their APIs, as they document them. A backtest is a comment, not a check that can stop a merge.
- A pull request is opened only for a file in the repository that is the document Lapilli read. A rule file mounted from a ConfigMap whose manifest is what git holds, rules rendered from a chart, and rules read back from a cluster get a diff for a person instead.
This page is copied from docs/limits.md, where it is changed: a pull request there changes it here.
Start here
Judging a period
- How it judges
- Use
lapilli verdict - The numbers
- A cloud's own alarms
- Rules on logs: Loki
- Is the replay right?
- What an alert says
Before a change
Incidents
Reports
Every hour: lapilli serve
Reference
And
Also at lapilli.dev/docs.