TwentyYearsSRE
title: Twenty Years of SRE Lessons Learned type: technique created: 2026-05-21 last_updated: 2026-05-21 related: ["radar/techniques/ServiceLevelObjectives", "radar/techniques/OpenTelemetryGitPerformance"] sources: ["https://sre.google/resources/practices-and-processes/twenty-years-of-sre-lessons-learned/"] radar_quadrant: Techniques radar_ring: Assess radar_position: inner
Google SRE's distilled practitioner lessons from two decades of production operations, published as a canonical reference for reliability engineering practice.
Toil is the enemy of reliability. Toil — manual, repetitive, automatable operational work — does not scale with system growth. Google's SRE model caps toil at 50% of a team's time; the rest goes to engineering work that eliminates future toil. Teams that allow toil to exceed this threshold drift toward pure operations and lose the capacity to improve reliability.
Error budgets make reliability negotiable. An error budget converts an SLO into a finite resource that both development and operations consume. When the budget is healthy, development moves fast. When it is exhausted, reliability work takes priority. This transforms a chronic tension between velocity and stability into a shared, quantifiable constraint.
Gradual rollouts eliminate a class of incidents. Canarying — releasing to 1% of traffic before 100% — catches configuration and code errors before they affect all users. The cost is rollout latency. The benefit is that the majority of production incidents caused by deployments are detectable and reversible before full exposure.
Postmortems are blame-free or useless. A postmortem that assigns blame to individuals suppresses the information needed to fix systemic causes. Blameless postmortems treat incidents as system failures, not human failures, and focus on eliminating the conditions that made the error possible.
On-call load predicts retention. High on-call frequency correlates with attrition. Google's model targets a maximum of two on-call incidents per 12-hour shift. Above this threshold, on-call engineers cannot investigate properly and carry over stress. Tracking on-call load as a reliability metric is as important as tracking SLO compliance.
Runbooks decay; automation does not. Runbooks require ongoing maintenance to remain accurate. Automated remediation runs the same code path every time and is self-verifying. The lesson is not to eliminate runbooks but to treat them as a temporary measure while building automation.
The twenty-year summary grounds the abstract radar/techniques/ServiceLevelObjectives methodology in operational experience. Error budgets, toil measurement, and on-call load tracking all presuppose a functioning SLO framework. The two blips are complementary: SLOs define the targets; the SRE lessons describe the organisational practices needed to act on them.
Twenty Years of SRE Lessons Learned sits in the Assess ring of the Techniques quadrant, at inner position. Published by Google SRE (2023-12-01 bookmark). These lessons are distilled from two decades of production operations at Google scale and validated by the broader SRE community since the original SRE Book (2016). That standing is third-party evidence, not first-person use: the principles have not been applied in the user's own operations, and the Adopt ring requires that, so the blip stays at Assess. This entry is closer to absorbed knowledge than to a discrete practice one trials; Assess holds it as a reference until the principles are applied first-person. Inner position reflects broad relevance to any production engineering team.
