Availability Group alerts: failover, replica disconnect, sync fell behind, database suspended (#991)

[erikdarlingdata]

Follow-up to #1688, which landed AG collection and explicitly deferred the alerts. This adds the alert family over those two tables.

The alerts

Metric name	Severity	Trigger
AG Failover	Warning	role_desc changed vs the previous sweep, per ag+replica
AG Replica Disconnected	Critical	connected_state_desc transitions to DISCONNECTED
AG Replica Reconnected	resolved (green)	recovery of the above
AG Sync Fell Behind	Warning	secondary_lag_seconds >= ag_lag_alert_seconds, or redo_queue_size >= ag_redo_queue_alert_kb
AG Database Suspended	Warning	is_suspended false -> true, detail carries suspend_reason_desc

Metric names are webhook automation keys and are consts, not inline literals. All five are registered in the shared AlertSeverity map so a history replay that reaches the map without an explicit override does not render INFO-blue.

Settings are store-backed on config.config_alert_settings (migration V35, all NOT NULL DEFAULT): notify_ag_health (true), ag_lag_alert_seconds (300, clamped 0-86400), ag_redo_queue_alert_kb (0 = off, clamped 0-1073741824), wired through DarlingConfig / DarlingAlertSettings / StoreConfigProvider / ViewerDataService / the viewer Settings window on the #1674 pattern and read live.

Judgment calls

State is keyed per AG grain, but alerts fire under the real server_id. A server hosts many replicas and databases, so two lagging databases must track and recover independently. Keying the alert that way would have cost the per-server delivery-mode override (#1236) and history correlation, so the grain lives in the alert text instead.

JudgeAgSync returns three states, not a bool. MS Learn documents secondary_lag_seconds reading 0 (not NULL) while data movement is SUSPENDED, so the seconds trigger has to abstain on a suspended row. Abstaining had to become its own answer rather than a plain "not behind", or the caller reads it as recovery: a lagging database that got suspended, or whose columns went NULL under quorum loss, would announce "AG Sync Recovered - has caught up with the primary" in the same sweep that reported it suspended. Only a database a sweep measured as caught up resolves a standing alert. That also makes cross-server resolution structurally impossible rather than something a prefix check has to catch. The redo-queue trigger keeps judging while suspended, because that backlog is real.

One statement per command. The two grains were briefly read as two statements over a single command to save a round trip. That cannot work: Npgsql only splits multi-statement text into a batch when it parses the SQL for named placeholders. With the positional ($1) parameters every read in this file uses, it sends one extended-protocol Parse and PostgreSQL rejects it with cannot insert multiple commands into a prepared statement. Because the path is failure-isolated, the whole family would have been silently dead behind one logged error per server per sweep. Split into two commands, each gated on its own snapshot time, which is independently correct: the two AG collectors are scheduled separately, so a healthy replica-grain timestamp must not vouch for stale database-grain rows. The read is skipped entirely when the master switch is off, so an AG-free fleet pays nothing.

NULL is never a transition. Under WSFC quorum loss the AG catalog views serve only cached metadata and any column can read NULL. Treating that as an edge would spray alerts across the fleet at the exact moment the cluster is already in trouble.

Adjacent drift fixed in the same change

• AlertMetricClassifier.IsResolution recognized only Cleared/Resolved/Restored. Darling's Collection Resumed, Agent Restarted and Compression Job Recovered recoveries have been shipping styled as live actionable alerts in both apps' Alert History grids. Widened to include Resumed/Restarted/Recovered/Reconnected rather than letting the AG recoveries become a fifth blind spot; no actionable metric name in either app contains those words. Both hand-maintained SQL copies of that list (Darling DailySummarySql, Lite LocalDataService.DailySummary) are widened to match, since they drive the Daily Summary actionable-alert counts in both apps.
• get_alert_settings (MCP) stopped at 36 columns while the store grew to 41, so an MCP client could not see the V33 connection opt-ins at all. Extended to 41. ViewerControlPlaneStage3bTests, the parity test pinning the column list against the upsert placeholders, bind order and reader ordinals (the guard for the riskiest defect class in this plumbing), had drifted the same way; extended, and its placeholder loop now runs off Columns.Length so the literal cannot drift again.
• A real ~1-in-6 intermittent suite failure, and it was not this work. ViewerTimeHelper keeps the display mode and UTC offset in process-wide statics; three test classes must mutate them, and their shared viewer-time-statics collection had no CollectionDefinition, so it grouped its members without constraining them and xUnit ran it in parallel with every other collection. Any viewer test rendering a timestamp could observe a swapped mode mid-assertion, and the victim differed run to run (ViewerSystemEventsTests, then ViewerWave3DisplayTests) - the shape that reads as "flaky, re-run it". Added the definition with DisableParallelization, constraining the three mutators instead of chasing an open-ended set of readers. Verified with 12 consecutive full-suite runs.
• TestResults/ is now gitignored.

The two migration pin tests are re-pinned by identity rather than by ordinal and literal count - the same fact stated twice made every stacked branch collide on them.

Known limits (deliberate, not oversights)

• No Lite parity. Lite collects both AG grains but gets no AG alerts; this was scoped to the Darling service. JudgeAgSync is a pure static and is the obvious PerformanceMonitor.Common twin of ConnectionAlertPolicy whenever Lite follows.
• No cross-node de-duplication. Every replica in an AG is visible from every node, so a 3-node AG with all 3 nodes monitored reports each role change three times, once per monitored server. Filtering to the local replica would need is_local on the replica grain, which the collector does not carry. Consistent with how every other alert in the product reports each server's own view, but worth a follow-up if it proves noisy in the field.
• AG Replica Disconnected is a pure edge with no re-fire, matching the connect edge's pre-Connection alerts never fire for a server that is already down, and never re-fire during a standing outage #1659 behavior. The Connection alerts never fire for a server that is already down, and never re-fire during a standing outage #1659 re-fire treatment is the obvious follow-up if a standing AG outage needs re-announcing.
• AlertMetricClassifier.IsCritical still matches only Deadlock/Poison, so AG Replica Disconnected styles as a warning row in the history grids, exactly as Server Unreachable and Capture Down already do. Left alone: changing it would restyle existing alerts, which is a separate call.

Testing

dotnet test green on all three suites: Darling 3194, Lite 1494, Dashboard 768 (0 failed). No live Installer.Tests categories were run. 24 new test methods plus 12 new AlertMetricClassifier cases cover every transition including baselines, recoveries, NULL readings, the suspend/lag interaction, per-database independence, cross-server scoping, live threshold re-reads, and Forget.

Reviewed by a code-reviewer subagent before push; its two highest findings (the multi-statement read and the false-recovery path) are the two fixed above.

🤖 Generated with Claude Code