Alert on a suspended secondary that is falling behind (#991)

[erikdarlingdata]

Corrects a rule I shipped in #1692 that silenced the alert on the most common way a secondary falls behind.

What was wrong

The lag trigger made the seconds check abstain on a suspended row. That was written to MS Learn, which states secondary_lag_seconds "shows as 0 if the data movement is suspended" -- so abstaining looked like the careful choice, since a zero would otherwise read as "caught up" on the database that is furthest behind.

The documentation is wrong. ag-fixture-builder measured the inverse on the live Docker AG fixture, ag-collector-builder confirmed it, and I re-ran the suspend/resume cycle myself before changing shipped logic (SQL Server 2022 16.0.4265.3, clusterless AG, write load, sampled across a SUSPEND_FROM_USER on the secondary):

is_suspended=1  lag=3993  redo=0  send=NULL  NOT SYNCHRONIZING
is_suspended=1  lag=4005  redo=0  send=NULL  NOT SYNCHRONIZING
is_suspended=1  lag=4017  redo=0  send=NULL  NOT SYNCHRONIZING
is_suspended=1  lag=4029  redo=0  send=NULL  NOT SYNCHRONIZING
is_suspended=1  lag=4041  redo=0  send=NULL  NOT SYNCHRONIZING
after RESUME :  lag=0     redo=0  send=0     SYNCHRONIZED

+48 seconds of lag across four 12-second intervals -- it accrues at wall-clock rate and returns to 0 on resume. So the abstention was not being careful, it was silencing the alert on suspended data movement. A suspended secondary could drift arbitrarily far behind while only AG Database Suspended fired once, on the edge.

The fix

A suspended row may raise an alarm but may never clear one.

This is deliberately correct under both behaviors rather than betting on the measurement:

• if lag accrues (measured), a suspended secondary crosses the threshold and fires;
• if it ever did read 0 (documented), that 0 is under the threshold and yields NotMeasurable rather than CaughtUp, so it still cannot resolve a standing alert -- which is the false-recovery bug the three-state judgement was introduced for in the first place.

The same rule now also protects the redo trigger, whose value freezes at its last reading while suspended (also measured). Frozen and over the threshold is a real backlog worth firing on; frozen and under it is stale data that must not clear anything. Previously a small frozen redo queue could resolve a standing alert.

Documented, not coded

• log_send_queue_size reads NULL while suspended rather than growing, so it is useless as a fell-behind signal. This evaluator never used it -- noted so nobody reaches for it later.
• On resume the secondary has a real backlog to drain (388,620 KB after a 60-second suspend under load), so a single-sample redo threshold fires during legitimate catch-up. That is not wrong -- the data-loss window is genuinely open until the queue drains -- but it is why the redo trigger ships off, and why a consecutive-sample qualifier is the natural follow-up if it proves noisy.
• Vantage: sys.dm_hadr_* on a secondary carries only that replica's own rows, so a monitored secondary sees a one-row self-view of its AG. Nothing here assumed otherwise (the rules judge whatever rows arrive, keyed per ag+replica), but full AG coverage needs the primary monitored.

Not taken

The fixture also validated that replica-grain synchronization_health_desc flips HEALTHY -> NOT_HEALTHY during a suspend, which is a usable signal. I did not add a fifth alert for it -- it would largely duplicate AG Database Suspended, and the family was scoped to four. Worth considering as its own follow-up.

Testing

Darling suite green: 3252 passed, 0 failed. The pure judgement gains a case for every quadrant (suspended over/under threshold, for both triggers, plus the resumed measurement that does resolve), and there is a new end-to-end test for the exact case the old rule silenced: a suspended secondary drifting past the threshold now fires, stays quiet inside the cooldown, and resolves only once movement is running again.

The fixture was left SYNCHRONIZED and healthy.

🤖 Generated with Claude Code