Skip to content

Burn-rate alert summary claims 100% budget consumed when the standard says 10% #199

Description

@stxkxs

dashboards/base/alerting/agent-operator.yaml:178

summary: operator reconcile latency budget burning (100% over 3d)

This is the factor-1 / 3d-long tier (expr at :196 uses > bool 1). Burn rate 1 spends the error budget at exactly the rate that exhausts it over the 30-day SLO window, so three days of it consumes 3/30 = 10%, not 100%.

Two independent sources in the repo already say so:

  • nanohype/standards/observability-slo.json:62"budget_consumed": "10% in 3d" for this exact window
  • the rule's own description, one line below its summary"the budget is being spent at exactly the rate that exhausts it over the 30d SLO window"

The three sibling tiers in the same file all quote the standard correctly, which is what makes this a typo rather than a different reading:

tier summary standard
1h, factor 14 2% in 1h 2% in 1h ✅
6h, factor 6 5% in 6h 5% in 6h ✅
1d, factor 3 10% in 1d 10% in 1d ✅
3d, factor 1 100% over 3d 10% in 3d

Why it matters more than a wording nit

The summary is the alert title — it is what reaches a human first and often all they read. This one is severity: ticket, the least urgent tier, and its title describes total budget exhaustion, the most urgent condition there is.

Both outcomes are bad. Escalated, it reads as an outage that is not happening. Recognised as wrong, it teaches on-call that these titles are not to be trusted, which devalues the three tiers that are accurate.

Why no gate sees it

scripts/check-alert-coverage.py checks that alerts exist for the SLOs, not that their prose agrees with the standard's figures. The summary is a free-text annotation; nothing parses it. The expression is correct — only the sentence describing it is wrong, so every rule-level validation passes.

A gate is plausible here and cheap: the standard already carries budget_consumed per tier as structured data, so a check could assert the string appears in the matching rule's summary. That is the same shape as the existing check-athena-panel-columns.py — compare prose against the structured source it claims to describe.

Found by an adversarial sweep for instances of the failure classes in nanohype/docsHow this fails (class: a value whose report is not the value it measures).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions