-
Notifications
You must be signed in to change notification settings - Fork 0
Concepts Escalation Policies
An Escalation Policy defines what happens when an incident goes unacknowledged. Without one, an alert fires once and — if nobody responds — nothing else happens. An escalation policy solves this by sending notifications in a series of ordered steps, each targeting a different On-Call Schedule, until someone acknowledges the incident.
When an incident is created, Piro begins evaluating the escalation policy. A background job (running every minute) checks each open, unacknowledged incident against the policy steps:
- Step 0 fires immediately, or after its configured delay from the moment the incident was created.
- Step 1 fires if the incident is still unacknowledged after step 1's delay has elapsed since step 0 dispatched.
- Each subsequent step follows the same pattern — the delay is counted from when the previous step was sent, not from incident creation.
As each step fires, Piro resolves who is currently on-call in the step's referenced schedule and notifies them.
| Field | Description |
|---|---|
| Name | A descriptive label (e.g. "Default Policy", "P1 Response") |
| Description | Optional notes |
| Re-escalate after ACK (minutes) | If the incident is acknowledged but not resolved, re-start the escalation after this many minutes. Set to 0 to disable |
| Re-escalate after inactivity (minutes) | If there has been no human activity on the incident for this long, re-escalate. Set to 0 to disable |
Each step is evaluated in order (by Order field, ascending).
| Field | Description |
|---|---|
| Order | Position in the escalation sequence (0-based). Step 0 is always first |
| Delay (minutes) | For step 0: minutes from incident creation before the first notification. For step 1+: minutes from the previous step's dispatch time |
| Schedule | The On-Call Schedule to consult for who is currently on-call |
Example sequence:
| Step | Delay | Schedule | Effect |
|---|---|---|---|
| 0 | 0 | Backend On-Call | Notify immediately when incident opens |
| 1 | 10 | Backend On-Call | Notify again if still unacknowledged after 10 min |
| 2 | 20 | Engineering Leads | Escalate to leads if still unacknowledged 20 min after step 1 |
Two independent re-escalation thresholds let you handle incidents that stall after an initial acknowledgment:
- After ACK: someone acknowledged the incident but it never resolved. After the configured number of minutes, the escalation sequence restarts from step 0.
- After inactivity: no one has interacted with the incident (no comments, no status changes) for the configured number of minutes. Useful for catching incidents that were acknowledged but then forgotten.
Setting either value to 0 disables that re-escalation trigger entirely.
Each Service has its own optional escalation policy (Service.EscalationPolicyId). This means different services can escalate through entirely different chains — for example, a payments service escalating straight to on-call leads, while an internal tool escalates to a single Slack-adjacent channel via a lighter policy.
A service is not required to have a policy assigned. If it doesn't, alerts on that service's checks still fire and notify configured Triggers, but they will never escalate to an on-call schedule. The admin panel surfaces this with a warning (tooltip) on the service so it isn't missed silently.
⚠️ Note: If a service has no escalation policy assigned, its alerts will not escalate to on-call — they rely entirely on whatever triggers are attached directly to the alert config.
Admin panel → Services → select a service → Escalation Policy
Assign one of your existing policies to the service, or leave it unset. Policies themselves (name, re-escalation thresholds, steps) are managed from:
Admin panel → On-Call → Escalation Policies
From there you can create new policies, set re-escalation thresholds, and add or reorder steps — then assign the resulting policy to one or more services.
- Set step 0's delay to
0so the first on-call user is notified the moment an incident opens. Only add a delay if you want a brief grace period before paging anyone (e.g. to avoid noise from transient flaps). - Use the failure threshold on your Checks to avoid creating incidents from a single blip. Escalation policies work best when the incident itself is already meaningful.
- Configure Re-escalate after ACK for critical services. An acknowledgment means someone saw the alert — it does not mean the problem is resolved.
- Keep escalation chains short (2–3 steps). Long chains with many steps can create alert fatigue and delay getting the right person involved.
- Services — each service optionally references one escalation policy
- On-Call Schedules — each escalation step references a schedule to resolve the current on-call user
- Checks — failure thresholds control when an incident is actually created
- Triggers — notification channels used to deliver the escalation notifications