Skip to content

Spec Operations

mavrovde edited this page Nov 1, 2026 · 1 revision

Spec: Operations

Part of the Technical Specification. Requirement prefix: OPS. Method: the Google SRE practice of SLOs, error budgets, symptom-based alerting, runbooks and blameless postmortems. Targets come from Spec: Non-Functional Requirements.

1. SLIs and SLOs

SLI Definition SLO (design target) Window
Availability good = valid requests not answered 5xx / all valid requests 99.5 % rolling 30 days
Latency good = valid requests within their class's p99 target / all valid requests 99 % rolling 30 days
Licence integrity licence-gate post-check violations 0 always
Freshness reporting responses whose meta.freshness is complete 100 % rolling 30 days

2. Error budget policy

Budget remaining Action
> 50 % Normal delivery
25–50 % Releases continue; reliability work prioritised in planning
0–25 % Only fixes and reliability work are released; a review of the burn
Exhausted Feature releases frozen until the budget recovers or the owner decides otherwise in writing; postmortem required

3. Alerting

Alert on symptoms that hurt clients, using multi-window, multi-burn-rate rules.

Alert Condition (design) Severity
Fast burn availability burn rate ≥ 14.4 over 1 h and over 5 min page
Slow burn burn rate ≥ 6 over 6 h and over 30 min page
Gradual burn burn rate ≥ 1 over 3 days ticket
Licence-gate violation count > 0 page
Readiness failing /readyz failing for 5 min page
Deadline overrun any request observed above 2 s + margin ticket
Storage near its role limits permits saturated for 10 min ticket
Usage anomaly one key above 5× its 7-day hourly median for 1 h ticket
Certificate expiry under 14 days ticket
Unknown storage codes rising for 1 h ticket

4. Runbook template

# Runbook: <alert name>

- Alert: <expression and thresholds>
- Severity: page | ticket
- Requirement ids: <for example NFR-AVAIL-001>
- Impact: what clients experience
- Dashboards: <links>
- Diagnosis: ordered checks, each with the command or query and the expected result
- Mitigation: ordered actions, safest first (roll back, shed, disable an operation, raise a limit)
- Escalation: who, when
- Verification: how to confirm recovery
- Follow-up: postmortem needed? issue template

5. Incident management

flowchart LR
  D[Detect: alert or report] --> T[Triage and declare severity]
  T --> M[Mitigate first]
  M --> C[Communicate on the status page]
  C --> R[Resolve]
  R --> P[Postmortem within 5 working days]
  P --> A[Action items tracked to done]
Loading
Severity Definition Status page update Postmortem
SEV1 API unavailable or a licence or security breach within 30 min, then hourly required
SEV2 Major degradation of an endpoint class or a tier within 1 h required
SEV3 Minor degradation, workaround exists as needed optional

6. Postmortem template (blameless)

# Postmortem: <title>

- Date, duration, severity, authors
- Summary: what happened, in three sentences
- Impact: clients, endpoints, tiers, error budget consumed
- Timeline (UTC): detection, key decisions, mitigation, resolution
- Root cause: the chain of causes, not a person
- What went well, what went badly, where we were lucky
- Action items: each with owner, issue link, due date, and the requirement or test it adds

7. On-call (generic)

Stated generically because the team is small and the hours are not yet decided:

  • An on-call rota covers the hours the per-tier SLA promises (Spec: Developer Experience); outside them, alerts queue.
  • Acknowledge a page within 15 minutes inside covered hours.
  • The person on call may roll back, shed load, disable an operation or revoke a key without further approval.
  • Handover notes at the end of each rotation.

Requirements

ID Requirement Verification
OPS-001 SLIs MUST be computed from the service's own metrics, excluding 4xx and 429. Ops
OPS-002 The error budget policy MUST be applied as written; exceptions are recorded with the owner's decision. Review
OPS-003 Alerts MUST be symptom-based with multi-window burn rates; cause-based signals are tickets, not pages. Review
OPS-004 Every alert MUST link to a runbook in the template above. Review
OPS-005 Incidents MUST be declared with a severity, communicated on the status page within the stated time, and SEV1 and SEV2 MUST get a blameless postmortem within 5 working days. Ops
OPS-006 Postmortem action items MUST be tracked as issues with owners and due dates. Ops
OPS-007 Backups of the access store MUST run daily, be encrypted, and a restore MUST be tested at least quarterly; recovery point objective 24 h, recovery time objective 4 h (design targets). Ops
OPS-008 Capacity MUST be reviewed monthly against NFR-CAP; a forecast breach within 90 days opens an issue. Ops
OPS-009 A load test MUST NEVER be pointed at production. Review
OPS-010 The operator MUST be able to disable an operation, a source family or a key within 5 minutes without a deployment. Config
OPS-011 TLS certificates MUST renew automatically and be monitored for expiry. Ops
OPS-012 Per-key usage anomalies (section 3) MUST raise a ticket, as a signal of a leaked key. Ops
OPS-013 Time on every host and container MUST be synchronised (NTP), because quota resets, cursors and freshness depend on it. Ops

Home

About

Design

Data Model

Technical specification

Testing

Engineering

Technology choice

Reference

Clone this wiki locally