Skip to content

feat(monitoring): add backend-listen WS connection drop alert#9565

Merged
beastoin merged 1 commit into
mainfrom
fix/backend-listen-ws-drop-alert
Jul 12, 2026
Merged

feat(monitoring): add backend-listen WS connection drop alert#9565
beastoin merged 1 commit into
mainfrom
fix/backend-listen-ws-drop-alert

Conversation

@beastoin

@beastoin beastoin commented Jul 12, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds a critical alert for backend-listen active WebSocket connections dropping below 10. This catches any backend-listen failure that causes client disconnections, regardless of root cause (e.g., helm empty defaults incident).

Alert specification

Field Value
UID tz_backend_listen_ws_drop
Expression sum(backend_listen_active_ws_connections{job="backend-listen-metrics"}) or vector(0)
Threshold < 10
Duration 5m
Severity critical
Dashboard link Omi Services Overview → Panel 82 ("Backend-listen active WS")
instatus_component live-transcription

Baseline (3-day)

  • Min non-zero: 52
  • Average: 218
  • Max: 365
  • Threshold (10) is ~5% of minimum, giving wide margin before false positives

Expression alignment

Query matches dashboard panel 82 exactly: sum(backend_listen_active_ws_connections{job="backend-listen-metrics"})

Test plan

  • JSON syntax validation: traffic-zero.json passes json.load()
  • Dashboard annotation: panel 82 resolves in Omi Services Overview
  • Expression matches dashboard panel 82
  • Deploy via grizzly apply and verify alert evaluates to OK (mon)

🤖 Generated with Claude Code

Review in cubic

Fires when sum(backend_listen_active_ws_connections) < 10 for 5m.
Matches dashboard panel 82 expression. 3-day baseline: min=52,
avg=218, max=365. Catches any backend-listen failure that causes
client disconnections regardless of root cause.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@beastoin

Copy link
Copy Markdown
Collaborator Author

Confirmed, this approach is sound. Panel 82 is valid under omi-services-overview.json on dashboard uid 6d3d0897-c5a8-4f7b-b034-a3718c52a63b, and the or vector(0) behavior matches the existing traffic-zero rules where metric absence should page the same as zero traffic. The <10 threshold with for: 5m also lines up with the ops-specified 3-day baseline, so I do not see a remaining blocker here.


by AI for @beastoin

@beastoin

Copy link
Copy Markdown
Collaborator Author

All checkpoints passed — PR ready for merge

Checkpoint Status
CP0-CP3 ✅ (preflight, issue, workspace, exploration)
CP4 CODEx ✅ (2 turns — threshold/vector(0)/panel verified)
CP5-CP6 ✅ (implemented, PR created)
CP7 Review ✅ (PR_APPROVED_LGTM — JSON valid, PromQL correct, panel 82 resolves)
CP8 Tests ✅ (TESTS_APPROVED — config validation suite)
CP9A/B ✅ (JSON + annotation verification)

Awaiting manager merge approval.

by AI for @beastoin

@beastoin beastoin merged commit 56c6beb into main Jul 12, 2026
23 of 26 checks passed
@beastoin beastoin deleted the fix/backend-listen-ws-drop-alert branch July 12, 2026 13:08
@beastoin

Copy link
Copy Markdown
Collaborator Author

lgtm

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant