Skip to content

Incidents nobody wrote down

github-actions[bot] edited this page Nov 18, 2026 · 1 revision

The incidents a team writes down are the ones it noticed; those it didn't are the ones no alert caught. --impact finds them in a signal of user impact. A per-service error ratio from what is in front of your services works well, with --match service and labels renamed to service where needed (label_replace):

In front --impact
Istio sum by (destination_service_name) (rate(istio_requests_total{reporter="destination", response_code=~"5.."}[5m])) / sum by (destination_service_name) (rate(istio_requests_total{reporter="destination"}[5m])) > 0.01
Linkerd sum by (deployment) (rate(response_total{direction="inbound", classification="failure"}[5m])) / sum by (deployment) (rate(response_total{direction="inbound"}[5m])) > 0.01
Envoy, Envoy Gateway sum by (envoy_cluster_name) (rate(envoy_cluster_upstream_rq_xx{envoy_response_code_class="5"}[5m])) / sum by (envoy_cluster_name) (rate(envoy_cluster_upstream_rq_xx[5m])) > 0.01
ingress-nginx sum by (ingress) (rate(nginx_ingress_controller_requests{status=~"5.."}[5m])) / sum by (ingress) (rate(nginx_ingress_controller_requests[5m])) > 0.01
Traefik sum by (service) (rate(traefik_service_requests_total{code=~"5.."}[5m])) / sum by (service) (rate(traefik_service_requests_total[5m])) > 0.01
OpenTelemetry HTTP server metrics sum by (service_name) (rate(http_server_request_duration_seconds_count{http_response_status_code=~"5.."}[5m])) / sum by (service_name) (rate(http_server_request_duration_seconds_count[5m])) > 0.01

Label names depend on how each is scraped (Linkerd's deployment, an OpenTelemetry service's service_name or job): check them against your own Prometheus first. Pick the threshold your users would notice, and --impact-for long enough that a blip isn't an incident.


This page is copied from docs/impact.md, where it is changed: a pull request there changes it here.

Clone this wiki locally