fix(monitoring): resolve 5XX alert false positives from Stackdriver no-data

[beastoin]

Summary

• Fix 3 Stackdriver-based 5XX alert rules that fire false positives when there are zero 5XX errors
• Change noDataState from "Alerting" to "OK" for backend, backend-integration, and backend-sync 5XX rules

Root Cause

Stackdriver queries filtered by response_code_class = "500" return no data when there are zero 5XX errors. With noDataState: "Alerting", Grafana treats no-data as an alert condition and fires with sentinel value -1. This is the firing alert mon reported: "Backend - 5XX error rate is high" with value -1.

Prometheus-based 5XX rules are not affected because they use or vector(0) to return 0 instead of no-data. Stackdriver queries cannot use or vector(0).

Changes

File	UID	Rule Title
backend.json	cew4jcnpa68sga	Backend - 5XX error rate is high
backend-integration.json	eew96o25qztvkf	Backend-integration - 5XX error rate is high
backend-sync.json	cew97uqu791q8a	Backend-sync - 5XX error rate is high

Each: noDataState: "Alerting" → noDataState: "OK"

Test plan

• JSON validation passes (python3 -m json.tool on all 3 files)
• No remaining noDataState: "Alerting" in the 3 changed files
• Deploy to Grafana and verify the firing alert resolves
• Verify 5XX alerts still fire correctly when actual 5XX errors occur (non-empty Stackdriver response)

🤖 Generated with Claude Code

[beastoin]

@beastoin No issues found. The diff is limited to the three Stackdriver-backed 5XX rules, and noDataState: "OK" is the right behavior for these response_code_class = "500" queries because zero 5XX traffic can legitimately produce no series; execErrState: "Error" remains in place, so datasource/query failures still alert, and real non-empty 5XX series still flow through the existing reduce/threshold checks. Verified with python3 -m json.tool on all three changed alert files, rg confirming no remaining noDataState: "Alerting" in those files, and jq inspection of the three UIDs/filters/thresholds. PR_APPROVED_LGTM

Please keep the deploy-side Grafana resolution check from the PR test plan.

---

by AI for @beastoin

[beastoin]

Test results:

• python3 -m json.tool backend/charts/monitoring/alerts/backend.json >/dev/null — pass
• python3 -m json.tool backend/charts/monitoring/alerts/backend-sync.json >/dev/null — pass
• python3 -m json.tool backend/charts/monitoring/alerts/backend-integration.json >/dev/null — pass
• grep -n '"noDataState": "Alerting"' backend/charts/monitoring/alerts/backend.json backend/charts/monitoring/alerts/backend-sync.json backend/charts/monitoring/alerts/backend-integration.json — pass, no matches\n\nCoverage review: config-only change; JSON validation is the only executable test needed. Raw Alerting text still exists only in the unchanged receiver label Omi - Services Alerting (Telegram), not in noDataState.\n\nTESTS_APPROVED\n\n---\n_by AI for @beastoin_

[kodjima33]

Backend bug fix — root cause stated, scoped, CI green. Merging.