Repository navigation
Use lapilli verdict
# a week of your Grafana rules, against your own Prometheus
lapilli verdict --rules alert-rules.yaml --prometheus http://prometheus:9090 --incidents incidents.yaml
# what Prometheus actually fired last month
lapilli verdict --alerts --prometheus http://prometheus:9090 --since 30d --incidents incidents.yaml
# a rule you are about to merge: would it have caught last quarter's incidents, and how often would it have paged?
lapilli verdict --rules new-rule.yaml --prometheus http://prometheus:9090 --since 90d --incidents incidents.yaml
# weekly, into node_exporter's textfile collector
lapilli verdict --rules alert-rules.yaml --prometheus http://prometheus:9090 --incidents incidents.yaml \
--metrics > lapilli.prom.tmp && mv lapilli.prom.tmp lapilli.prom--json gives the whole report, every firing and its stone included;
--csv the firings alone, a row each (rule, severity, labels, start, end,
stone, the incidents it caught, re-fired), for a spreadsheet, a cell that
starts with =, +, - or @ written with ' before it so it isn't
run. docs/numbers.md defines every number, says how to
count each again from --json with jq, and lists the fields, columns and
metrics that keep their names. --metrics gives lapilli_rule_firings, lapilli_rule_caught,
lapilli_rule_nothing, lapilli_rule_refired, lapilli_rule_incidents,
lapilli_rule_incidents_first, lapilli_rule_incidents_alone (by rule),
lapilli_rule_nothing_by_hours (by rule and hours: asleep,
off_hours, at_work), lapilli_incidents (by state and source) and
lapilli_incident_detection_delay_seconds.
A replay asks for each query a day at a time, once for every rule that needs
it, and runs --parallel (8) rules at once against each API it reads:
--prometheus, --loki, each of Grafana's data sources; a Grafana is asked
--parallel requests at once at most for all its data sources, as it answers
for them all, shared fairly: a data source that holds places doesn't take the
last one while another holds none that asked within ten seconds, and a place
given back goes to the one waiting that holds fewest, so a slow data source
can't keep the others waiting on its answers. A server reached by two names,
as --loki and as a Grafana data source, or as --prometheus and --loki,
is asked as two. An API that turns a request away for now is asked again
after a wait:
- after HTTP 5xx (not 501), or a connection it dropped or reset, 2, 4, 8 and 16 seconds;
- after HTTP 429 (too many requests), or a 503, that says how long
(
Retry-After, in seconds or as a date), that long, between a second and two minutes, plus up to a quarter, as often as it says so; - after a 429 that doesn't, 4, 8, 16, 32, 60 and 60 seconds, a minute and a half to three minutes in all, past the window of a quota by the minute.
429s and 5xx are counted apart. Each wait is less up to half, or more up to
a quarter, so that requests turned away together don't come back together.
A request is asked again for five minutes at most from when it was first
pushed back, by an answer or by a wait for its turn the API asked for: the
API's answers and the waits for a turn count, and a request whose turn
doesn't come by then is given up; one already asked is waited for. A wait
that would end past --timeout, the API's or a turn by --max-rate, isn't
waited for: the rule is left unfinished then, as the deadline would leave
it, and the run exits 1 saying to raise --timeout.
A 429, a 503 or a connection dropped also halves how many requests may go
to that API at once: half of those in flight lately, or of the limit, if
fewer, from 64, for the whole run, down to one, once a second at most; eight
at once, as --parallel 8 has, are four after one push back. One more may
go for every ten seconds without a push back, asked or not, so lapilli serve, back an hour later, asks as many at once as before. While an API said
to wait, nothing else is asked of it: a request waiting for its turn looks
again when it wakes. Behind one Grafana, each data source backs off of its
own, as Grafana passes on what each answers. --max-rate N asks each API N
requests a second at most, evenly spaced, for a quota known beforehand:
--max-rate 0.5 for one of 30 a minute; a Grafana's data sources share it.
Runs at once that share a quota should each take their part of it. This is
the same for --prometheus and every cloud's --auth, for --loki, its
ruler included, and Grafana's data source proxy, and for lapilli serve,
which takes --max-rate too. --max-samples N (default 10,000,000) is the
most samples the answers to one read of a rule's query may hold in all, some
64 bytes each at the peak. A replay reads a day at a time, on days from
midnight UTC, so the bound is a day's: 1,440 samples a series at a minute,
some 6,900 series firing all day. Past it the rule isn't replayed, and the
reason says to read it at a longer --step, over a shorter --since, or with
a larger --max-samples. What lapilli does without, the configuration
and the targets Prometheus is asked for, and where a Loki's ruler runs its
groups as lapilli serve reads it each minute, is asked once, within 30
seconds, not while the API said to wait, and its answers don't change how
the rest is asked (the configuration 2 MiB at most, the targets 32, as
each is parsed whole); lapilli serve keeps where the groups run as last read
when it can't be read.
lapilli serve pointed at the history the alerts are evaluated against can
keep off their quota (docs/design/quota.md). Its
/metrics counts what each process asks of each API (prometheus, loki,
grafana): lapilli_serve_history_requests_total by the class of the
answer, the 429s, the requests asked again and the seconds waited, to set
beside the evaluator's own errors. Each of these is off by default:
--yield-on-429 stops a run's requests to an API at its first 429, not
asked again, and what reads the history after the run that round (reviews
of changes, the digest, backtests, changes carried out, the replay check)
waits for a later one; the rules left are replayed first by a later run,
from where they were missed, and the runs go no further than a week past
the oldest of them, so none is given up (--yield-for DUR, a day at most,
leaves the API alone longer). --guard EXPR holds each run, and each day of
a catch-up, while a PromQL expression such as
rate(prometheus_rule_evaluation_failures_total[5m]) > 0 returns anything
or can't be read; one refused as written fails the run. --catch-up-hours 01:00-06:00 keeps a catch-up, more than a day behind (the first start's
--backfill, a run after being down that long), to those hours in --tz,
at --catch-up-rate N if given. A run that waits is kept in the run history
as held, with why, and lapilli_serve_runs_held{reason} says it.
A run that takes longer than 15 seconds says on stderr, every 30 seconds,
how it is going; --progress=false keeps it quiet. A run the history
answered HTTP 429 says so at its end either way. stdout has the report
alone; the backtest recipes copy stderr to the job's log
as it comes, and none of the progress to the comment. A comparison counts
replays: a rule changed is replayed before and after.
lapilli verdict: after 2m30s, 23 of 61 rules replayed, 412 requests to the history, 37 asked again after a wait, 35 answered HTTP 429
A query fails one of two ways, and its rule's reason says which:
-
query failed: the history couldn't be read: ..., a failure of the run: the history wasn't reached (a connection, a name, TLS), didn't let lapilli in (HTTP 401, 403, 407, credentials that couldn't be had), wasn't where it was said to be (404, a redirect to sign in, a page or JSON that isn't the API's, from a proxy or a gateway), or turned the query away to the end (408, 429, 5xx). Another run, or this one with its access mended, may replay the rule. -
query failed: ...otherwise, the history's answer to the query as written, which every run gets: refused in the API's own words (HTTP 400 or 422: a parse error, an execution error), too large however short the range asked, not done (501), or an answer that can't be read.
A rule whose replay failed for the run (the history unread, or the time run
out) is named with why at the top of the report, as the rest of it leaves
the rule out, ten at most, and at its foot with the rules that can't be
replayed at all (a CloudWatch metric alarm, a rule paused, a query the API
refused as written). --json lists both in unreplayed, each with failed
true when its replay failed for the run and another run may replay it; in a
comparison, each with its side, before or after, but a rule alike in
both, replayed once for both, which has none. The exit stays 0, since what
was replayed holds; --fail-unreplayed exits 1 instead, for a check that a
run that couldn't read the history shouldn't pass; so too for a rule whose
labels on each of its series are more than --max-samples lets a replay
hold (too many labels on its series to replay), which a rule's writer
could do to keep it out of a backtest unseen. It doesn't count a query
refused as written, which fails every run until the rule changes, nor a
Grafana rule replayed whose query outside its condition failed, which the
report notes below its table. A run cut short by --timeout exits 1 either
way. The API's text in a reason is cut to 300 bytes, without what doesn't
show, and without the token, the signature or the password lapilli sent,
should the API echo it back; lapilli serve keeps a reason without how often
its request was asked.
--prometheus URL takes any Prometheus-compatible API: Prometheus, Thanos,
Mimir, VictoriaMetrics, or a cloud's managed Prometheus. --bearer-token-file
adds a token. For a cloud's, --auth sends the credentials it wants, taken
from those the machine or pod already has (environment, workload identity,
instance or managed identity, the cloud's CLI):
| Managed Prometheus | --prometheus |
--auth |
Read permission |
|---|---|---|---|
| Amazon Managed Service for Prometheus | https://aps-workspaces.REGION.amazonaws.com/workspaces/WORKSPACE |
aws (SigV4) |
aps:QueryMetrics |
| Azure Monitor managed service for Prometheus | the workspace's query endpoint, https://WORKSPACE.REGION.prometheus.monitor.azure.com
|
azure |
Monitoring Data Reader |
| Google Cloud Managed Service for Prometheus | https://monitoring.googleapis.com/v1/projects/PROJECT/location/global/prometheus |
google |
roles/monitoring.viewer |
| Amazon CloudWatch, its OpenTelemetry metrics | https://monitoring.REGION.amazonaws.com |
aws (SigV4) |
cloudwatch:GetMetricData, cloudwatch:ListMetrics
|
--aws-region names the region to sign for when the URL doesn't (a VPC
endpoint, say). --gcp-project PROJECT is short for Google Cloud's URL and
--auth google. Lapilli only reads from these.
CloudWatch answers PromQL over the metrics sent to its OpenTelemetry
endpoint, and over AWS's own metrics once their OpenTelemetry enrichment is
turned on. It answers at most 500 series a query and says so only in a
warning; Lapilli takes that as the query failing, so a rule replayed is never
one cut short. Its queries count against a daily allowance (100,000 series
and 300 million samples a day, by its documentation), so lower --parallel
and the period before replaying many rules over weeks. Google Cloud keeps
some of its services' metrics, Istio's and log-based metrics six weeks only,
and every point older than six weeks ten minutes apart, so a replay further
back than that is coarser there.
Oracle Cloud has no managed Prometheus to query: OCI Monitoring is queried in
its own language, MQL. On OKE, give --prometheus the Prometheus you run
there; it needs no --auth.
Each cloud keeps the alert rules of its managed Prometheus, and its own
alarms written in PromQL, in a form of its own, and --rules reads each:
| Cloud | Rules | To read them |
|---|---|---|
| AWS | a rule groups namespace | the AWS CLI's describe-rule-groups-namespace answer, as is; --loaded-rules --auth aws (aps:ListRules); or every namespace, as written, with --ruler https://aps.REGION.amazonaws.com/workspaces/ID --auth aws
|
| AWS | CloudWatch PromQL alarms | the AWS CLI's describe-alarms answer, as is. The pending period is for, the recovery period keep_firing_for. Its other alarms are listed as not replayed |
| Azure | Microsoft.AlertsManagement/prometheusRuleGroups |
an ARM template, or the JSON the Azure CLI or REST API gives for rule groups; or every rule group in a subscription or resource group with --ruler https://management.azure.com/subscriptions/SUB --auth azure. Severity 0 to 4 becomes severity=Sev0 to Sev4, and timeToResolve is how long an alert stays firing once its condition stops |
| Google Cloud |
Rules, ClusterRules, GlobalRules (monitoring.googleapis.com/v1) |
the resources, as files or from kubectl get ... -o yaml
|
| Google Cloud | Cloud Monitoring alerting policies with a PromQL condition |
gcloud monitoring policies list, in YAML or JSON, or the API's answer. The condition's duration is for, its alertRule the name; its severity becomes severity=critical, error or warning; an alert closes at the first evaluation at least the larger of 270 s and two evaluations after its condition last held, as Google Cloud closes them. Every policy's interval is a multiple of 30 s (30 s unless it says), so give --step 30s. Policies with other conditions are listed as not replayed |
| Oracle Cloud | no managed Prometheus | the rule files of the Prometheus you run |
Google Cloud evaluates Rules within their namespace and cluster, and
ClusterRules within their cluster; Azure, a rule group with a
clusterName within that cluster. Lapilli does the same where the resource
says how: every selector in a Rules expression gets its namespace as a
namespace matcher, and in an Azure rule group with a clusterName, that
as a cluster matcher. A Google Cloud cluster's name isn't in the
resource, so with several clusters in one project, add cluster matchers
to the expressions, or point --prometheus at that cluster's data alone.
This page is copied from docs/verdict.md, where it is changed: a pull request there changes it here.
Start here
Judging a period
- How it judges
- Use
lapilli verdict - The numbers
- A cloud's own alarms
- Rules on logs: Loki
- Is the replay right?
- What an alert says
Before a change
Incidents
Reports
Every hour: lapilli serve
Reference
And
Also at lapilli.dev/docs.