Skip to content

SLO-based alert routing (per-endpoint latency budgets) #299

Description

@YaronZaki
  • Difficulty: intermediate · Effort: M · Priority: P3
  • Impact: every alert
  • Problem Statement
    Today Sentry fires on every error regardless of priority. A single
    user's /api/save-bug-report typo blows up in Sentry the same way
    as a global outage of /api/check-user.
  • Why It Matters
    Alert fatigue desensitises operators. A read-endpoint 4xx is noise;
    an auth-mutation 5xx is a P0.
  • Expected Outcome
    Per-endpoint SLO burned-rate alert rules in Sentry:
    • Auth/mutation routes: 99% success budget.
    • Read routes: 95% success budget.
    • /health, /metrics: no alerts (probe routes).
  • Acceptance Criteria
    • Test synthesises a 1% error burst on /api/check-user; Sentry
      alert routes to #oncall-protocol (vs #oncall-fyi).
    • Read-route errors demote to info level.
  • Implementation Notes
    Sentry's Alerts UI: Burn Rate, Window 1h and 6h. Code-readable
    thresholds in quantara/web_app/utils/sentry_alerts.yaml.
  • Files / Modules Affected
    • new: quantara/web_app/utils/sentry_alerts.yaml
    • Sentry UI configuration
  • Dependencies: standalone.

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions