Repository navigation
Incidents nobody wrote down
The incidents a team writes down are the ones it noticed; those it didn't
are the ones no alert caught. --impact finds them in a signal of user
impact. A per-service error ratio from what is in front of your services
works well, with --match service and labels renamed to service where
needed (label_replace):
| In front | --impact |
|---|---|
| Istio | sum by (destination_service_name) (rate(istio_requests_total{reporter="destination", response_code=~"5.."}[5m])) / sum by (destination_service_name) (rate(istio_requests_total{reporter="destination"}[5m])) > 0.01 |
| Linkerd | sum by (deployment) (rate(response_total{direction="inbound", classification="failure"}[5m])) / sum by (deployment) (rate(response_total{direction="inbound"}[5m])) > 0.01 |
| Envoy, Envoy Gateway | sum by (envoy_cluster_name) (rate(envoy_cluster_upstream_rq_xx{envoy_response_code_class="5"}[5m])) / sum by (envoy_cluster_name) (rate(envoy_cluster_upstream_rq_xx[5m])) > 0.01 |
| ingress-nginx | sum by (ingress) (rate(nginx_ingress_controller_requests{status=~"5.."}[5m])) / sum by (ingress) (rate(nginx_ingress_controller_requests[5m])) > 0.01 |
| Traefik | sum by (service) (rate(traefik_service_requests_total{code=~"5.."}[5m])) / sum by (service) (rate(traefik_service_requests_total[5m])) > 0.01 |
| OpenTelemetry HTTP server metrics | sum by (service_name) (rate(http_server_request_duration_seconds_count{http_response_status_code=~"5.."}[5m])) / sum by (service_name) (rate(http_server_request_duration_seconds_count[5m])) > 0.01 |
Label names depend on how each is scraped (Linkerd's deployment, an
OpenTelemetry service's service_name or job): check them against your
own Prometheus first. Pick the threshold your users would notice, and
--impact-for long enough that a blip isn't an incident.
This page is copied from docs/impact.md, where it is changed: a pull request there changes it here.
Start here
Judging a period
- How it judges
- Use
lapilli verdict - The numbers
- A cloud's own alarms
- Rules on logs: Loki
- Is the replay right?
- What an alert says
Before a change
Incidents
Reports
Every hour: lapilli serve
Reference
And
Also at lapilli.dev/docs.