Add observability, service targets, backup/restore, and failure runbooks

[alexeygrigorev]

Parent epic: #8

Normative spec: 07 — Observability, service targets, backups, failure behavior
Decision dependency: #26

Scope

Implement structured redacted application/job/message logs, domain/availability/latency/error/freshness/queue/SES/scoring/auth/export/database/storage metrics, owner/threshold/runbook/escalation alerts, dashboard, deployed-version reporting, encrypted backup verification, restore/reconcile/tombstone/outbox suppression, immutable-image rollback, secret/provider expiry checks, and fault-degradation runbooks against approved SLO/RPO/RTO.

Non-goals

No raw email/submission/token logging, optional tracing dependency that can break startup, or untested backup-success assumption.

Acceptance criteria

• Every Decision: Approve service levels, recovery targets, and alert ownership #26 target has measurable query, owner, threshold/window, alert, runbook, and dashboard.
• Request/job/message/content/cohort/event correlation works with documented redaction/cardinality limits.
• GitHub/search/worker/SES/OIDC/database/edge failures degrade as spec and alert without losing committed state.
• Automated backup verification and restore drill meet approved RPO/RTO, retain active content, replay deletion tombstones, and suppress/reconcile historical outbox.
• Immutable-image rollback preserves new writes/idempotency and reports exact deployed SHA/digest.
• Credential/certificate/provider expiry is monitored before failure.

Test scenarios

1. Fault inject each external/dependency state and validate response, metric, alert, dashboard, and runbook action.
2. Restore snapshot containing old pending email and deleted user; reconcile before workers, prove no resend/resurrection.
3. Roll back application image after new registrations/enrollments and verify compatibility/no duplicate jobs.

No product Playwright surface is required beyond viewing operator dashboard in #32; deployed browser smoke and screenshot evidence is covered by #70/#76.

Dependencies

Depends on #26, #31, domain metrics, and infrastructure #67–#70.