sre-runbooks

Runbooks and opinionated SRE practice templates distilled from running production Kubernetes platforms: SLO definitions, error budget policy, blameless postmortems, and on-call hygiene.

These are the documents I wish every team had on day one. Copy them, strip what you don't need, and adapt the numbers to your own reliability targets.

Contents

runbooks/
  node-not-ready.md          Kubernetes node NotReady: triage, drain, replace
  slo-fast-burn.md           Error budget burning at 14.4x: stop it, then explain it
  _template.md               Skeleton for a new runbook
slo/
  slo-template.md            SLI/SLO definition worksheet with a worked example
  error-budget-policy.md     What happens when the budget burns
incidents/
  postmortem-template.md     Blameless postmortem structure
  incident-severity-matrix.md  SEV1-SEV4 with response expectations
oncall/
  oncall-handbook.md         Handoff checklist, escalation, alert hygiene
scripts/
  check-runbooks.py          Structure and link checks, run in CI

Runbooks

The runbooks pair with the alerting rules in k8s-monitoring-stack — each one is the document its alert's runbook_url should point at.

Every runbook answers the same four questions in the same order, because at 03:00 nobody wants to learn a new document layout: what is actually true when this fires, what to run in the first five minutes, how to stop the bleeding, and how to know it worked. Each one also says explicitly when the answer is wait — doing nothing is a legitimate step that runbooks systematically under-document, and treating every firing as an emergency is how people learn to skip the runbook entirely.

scripts/check-runbooks.py enforces that structure in CI: required header fields, the required sections in the required order, at least one runnable command, and every relative link and heading anchor resolving. External URLs are deliberately not fetched — a link checker that fails when somebody else's site is down is a link checker people start ignoring. The template is validated by the same script, so it cannot drift away from the rules it demonstrates.

python3 scripts/check-runbooks.py

Philosophy

• SLOs before dashboards. A dashboard nobody alerts on is decoration. Define the SLI, set the target, alert on burn rate.
• Blameless, not toothless. Postmortems name systems and gaps, never people — but every one ends with owned action items and due dates.
• Alerts must be actionable. If a page has no runbook link and no clear action, it gets deleted or demoted to a ticket.

License

MIT