POSTMORTEM

The SRE Engagement Model

What is and isn’t SRE?

SRE(Site or Software Reliability Engineering) is an engineering approach to solving reliability concerns for any ecosystem.

SRE increases the availability of services by fixing root causes(trivial bug, or architectural deficiencies or missing features), reducing human toil(repetitive work)

SREs have moral authority and responsibility to look over the landscape that they are in and have access to design artifacts or source so that they can dive deep and come up with solutions or drive the teams to come up with the solution.

If a system needs human operator, it has a bug. This needs to be fixed to reduce Toil.

SRE is not L1 support. SREs typically get involved when an incident is breaching the SLO of the ecosystem.

Availability - Availability, in SRE terms, defines whether a system is able to fulfill its intended function at a point in time

When we set out to define the terms of SRE, we wanted to set a precise numerical target for system availability. We term this target the availability Service-Level Objective (SLO) of our system. Define the lowest level of reliability that you can get away with for each service, and state that as your SLO.

This might be expressed in availability numbers: for instance, an availability SLO of 99.9% over one month, with an internal availability SLO of 99.95%

TOIL(https://landing.google.com/sre/workbook/chapters/eliminating-toil/)

SREs spend much of their time optimizing—squeezing every bit of performance from a system through project work and developer collaboration. But the scope of optimization isn’t limited to compute resources: it’s also important that SREs optimize how they spend their time. SREs should not spend more than X%(typically 50%) of their time on non-productive, repetitive work. They strive to fix the root causes leading to the repetitive work so that it saves time, improves the team morale etc.

Error budget (https://landing.google.com/sre/sre-book/chapters/embracing-risk/)

If a team breaches their SLO availability, they will not be allowed to push new changes until they fix the underlying root causes which caused the reduced availability.

Post mortems

Tesco Till SRE SLOs

Availability – 99.5% for sale txn

How do we measure – link to dashboard

Error budget – 0.5% or lesser per team

SLO – same as above

Toil – 50% or lesser

Post mortems – link to the post mortem doc

Quoting from https://landing.google.com/sre/ and https://landing.google.com/sre/sre-book/chapters/eliminating-toil/:

Like traditional operations groups, we keep important, revenue-critical systems up and running despite hurricanes, bandwidth outages, and configuration errors. Unlike traditional operations groups, we view software as the primary tool through which our systems are managed, maintained, and minded; to that end, we have the source-level access and moral authority required to fix, extend and scale code to keep it working, harden it against the vagaries of the Internet, and develop our own planet-scale platforms.

If a human operator needs to touch your system during normal operations, you have a bug. The definition of normal changes as your systems grow - Carla Geisser, Google SRE

"Site Reliability Engineering seeks to balance the risk of unavailability with the goals of rapid innovation and efficient service operations, so that users’ overall happiness—with features, service, and performance—is optimized." - Marc Alvidrez, Site Reliability Engineer, Mountain View

"SREs engineer services, instead of binaries. This is a shift in perspective that exploits unusual skills and creativity. SREs are specialists in making changes safely." - John T. Reese, Site Reliability Engineer, San Francisco

"Here’s what you do when someone breaks something or finds something very difficult to debug: You say thank you. Thank you for finding this edge case. Thank you for highlighting this overcomplicated part of our system. Thank you for pointing out this gap in our docs. And then you go make it so nobody can break it the same way again." - Tanya Reilly, Google SRE 2005 - 2018

"Fundamentally, it's what happens when you ask a software engineer to design an operations function." Ben Treynor Sloss, Vice President, Google Engineering, Founder of Google SRE

SRE seeks production responsibility for important services for which it can make concrete contributions to reliability. SRE is concerned with several aspects of a service, which are collectively referred to as production. These aspects include the following:

System architecture and inter-service dependencies

Instrumentation, metrics, and monitoring

Emergency response

Capacity planning

Change management

Performance: availability, latency, and efficiency

When SREs engage with a service, we aim to improve it along all of these axes, which makes managing production for the service easier.

In general, an SRE team is responsible for the availability, latency, performance, efficiency, change management, monitoring, emergency response, and capacity planning of their service(s). 

Priniciples of SRE

1. Operations is a software problem — “The basic tenet of SRE is that doing operations well is a software problem. SRE should therefore use software engineering approaches to solve that problem.”

2. Manage by Service Level Objectives (SLOs) — Maintaining 100% availability isn’t the goal of SRE. “Instead, the product team and the SRE team select an appropriate availability target for the service and its user base, and the service is managed to that SLO. Deciding on such a target requires strong collaboration from the business.”

3. Work to minimize toil — Toil is tedious, manual, work. SRE doesn’t accept toil as the default. “We believe that if a machine can perform a desired operation, then a machine often should. This is a distinction (and a value) not often seen in other organizations, where toil is the job, and that’s what you’re paying a person to do.”

4. Automate this year’s job away — Automation goes hand-in-hand with reducing toil by “determining what to automate, under what conditions, and how to automate it.”

5. Move fast by reducing the cost of failure — The later a problem is discovered, the harder it is to fix. SRE addresses this issue. “SREs are specifically charged with improving undesirably late problem discovery, yielding benefits for the company as a whole.”

6. Share ownership with developers — SRE aims to reduce boundaries. “Ideally, both product development and SRE teams should have a holistic view of the stack—the frontend, backend, libraries, storage, kernels, and physical machine—and no team should jealously own single components.”

7. Use the same tooling, regardless of function or job title — In SRE, you can’t have different teams using different sets of tools. “There is no good way to manage a service that has one tool for the SREs and another for the product developers, behaving differently (and potentially catastrophically so) in different situations. The more divergence you have, the less your company benefits from each effort to improve each individual tool.”

We’ll use this hierarchy, illustrated in Fig-1, to look at the elements that go into making a service reliable, from most basic to most advanced.

Fig 1 : Service Reliability Hierarchy

Monitoring

Analyzing long-term trends

Comparing over time or experiment groups

Alerting

Building dashboards

Incident Response

Effective incident management is key to limiting the disruption caused by an incident and restoring normal business operations as quickly as possible. If you haven’t gamed out your response to potential incidents in advance, principled incident management can go out the window in real-life situations.

Postmortem and Root-Cause Analysis

The primary goals of writing a postmortem are to ensure that the incident is documented, that all contributing root cause(s) are well understood, and, especially, that effective preventive actions are put in place to reduce the likelihood and/or impact of recurrence.

Writing a postmortem is not punishment—it is a learning opportunity for the entire company. The postmortem process does present an inherent cost in terms of time or effort, so we are deliberate in choosing when to write one. Teams have some internal flexibility, but common postmortem triggers include:

User-visible downtime or degradation beyond a certain threshold

Data loss of any kind

On-call engineer intervention (release rollback, rerouting of traffic, etc.)

A resolution time above some threshold

A monitoring failure (which usually implies manual incident discovery)

It is important to define postmortem criteria before an incident occurs so that everyone knows when a postmortem is necessary. In addition to these objective triggers, any stakeholder may request a postmortem for an event.

Blameless postmortems are a tenet of SRE culture. For a postmortem to be truly blameless, it must focus on identifying the contributing causes of the incident without indicting any individual or team for bad or inappropriate behavior. A blamelessly written postmortem assumes that everyone involved in an incident had good intentions and did the right thing with the information they had. If a culture of finger pointing and shaming individuals or teams for doing the "wrong" thing prevails, people will not bring issues to light for fear of punishment.

Testing

Once we understand what tends to go wrong, our next step is attempting to prevent it, because an ounce of prevention is worth a pound of cure. Test suites offer some assurance that our software isn’t making certain classes of errors before it’s released to production.

Capacity Planning

Capacity planning is work teams do to make sure their services have enough spare capacity to handle any likely increases in workload, and enough buffer capacity to absorb normal workload spikes, between planning iterations.

Development

Applying engineering to fix faults, rather than avoiding or minimizing them.

Product

Product should have these features.

Lightweight

Easy on developers

Robust

Catches obvious errors

Thorough

Addresses important details consistently and reproducibly

Postmortem issue fields

Field

Instructions

Incident summary

Summarize the incident in a few sentences. Include what the severity was, why, and how long impact lasted.

Leadup

Describe the circumstances that led to this incident, for example, prior changes that introduced latent bugs.

Fault

Describe what didn't work as expected. Attach screenshots of relevant graphs or data showing the fault.

Impact

Describe what internal and external customers saw during the incident. Include how many support cases were raised.

Detection

How and when incident was detected?

How could time to detection be improved? As a thought exercise, how would you have cut the time in half?

Response

Who responded, when and how? Were there any delays or barriers to our response?

Recovery

Describe how and when service was restored. How did you reach the point where you knew how to mitigate the impact?

Additional questions to ask, depending on the scenario: How could time to mitigation be improved? As a thought exercise, how would you have cut the time in half?

Timeline

Provide a detailed incident timeline, in chronological order, timestamped with timezone(s). 

Include any lead-up; start of impact; detection time; escalations, decisions, and changes; and end of impact.

Five whys

Use the root cause identification technique.

Start with the impact and ask why it happened and why it had the impact it did. Continue asking why until you arrive at the root cause.

Document your "whys" as a list here or in a diagram attached to the issue.

Root cause

What was the root cause? This is the thing that needs to change in order to stop this class of incident from recurring.

Backlog check

Is there anything on your backlog that would have prevented this or greatly reduced its impact? If so, why wasn't it done?

An honest assessment here helps clarify past decisions around priority and risk.

Recurrence

Has this incident (with the same root cause) occurred before? If so, why did it happen again?

Lessons learned

What have we learned?

Discuss what went well, what could have gone better, and where did we get lucky to find improvement opportunities.

Corrective actions

What are we going to do to make sure this class of incident doesn't happen again? Who will take the actions and by when? 

Create "Priority action" issue links to issues tracking each action. 

Timeline/SLA:

Complete the post-mortem within 1 week

Review with extended Till ecosystem and close the post-mortem in the 2nd week

Complete the identified corrective actions within 1 month.

Reference:

https://www.atlassian.com/incident-management/handbook/postmortems#what-is-post-mortem