From 9725a10040c04c0309c4b6cf4cbea0c61323e65d Mon Sep 17 00:00:00 2001 From: Erik Darling <2136037+erikdarlingdata@users.noreply.github.com> Date: Sun, 26 Jul 2026 16:11:27 -0400 Subject: [PATCH] State the commit-time conclusion the shipped guidance rests on #1708 corrected the AG lag guidance to "do not derive lag from commit times at all" and cites this file. The file carried both halves of that argument but in separate sections, never drawn together - so a reader who found one half could reasonably conclude the other case was safe to guard against, which is the exact mistake #1708 was fixing. Now stated side by side: on a SUSPENDED replica last_commit_time freezes, so now - last_commit_time stops growing exactly when replication stops (silent, understating at the moment it is worst); on a healthy but QUIET database nothing commits, so the same delta grows without bound (loud, measured 1757 seconds at zero real lag). Guarding only the suspended case does not make a commit-delta trigger safe - it converts a silent failure into a noisy one. The loud half is also the likelier to ship, since it appears the first time anyone points it at a database nobody is writing to. Evidence file and CHANGELOG only, no code. Co-Authored-By: Claude Fable 5 --- CHANGELOG.md | 2 ++ tools/ag-fixture/VALIDATION.md | 20 ++++++++++++++++++++ 2 files changed, 22 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index f3f4bcdc6..991293f51 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- **AG fixture evidence: state the commit-time conclusion the guidance rests on** ([#1709]) - [#1708] corrected the AG lag guidance to "do not derive lag from commit times at all", and cites `tools/ag-fixture/VALIDATION.md`. The file carried both halves of the argument but in separate sections, never drawn together - so the reader who found one half could reasonably conclude the other case was safe to guard against. It now states them side by side: on a **suspended** replica `last_commit_time` freezes, so `now - last_commit_time` stops growing exactly when replication stops (**silent**, understating at the moment it is worst); on a **healthy but quiet** database nothing commits, so the same delta grows without bound (**loud**, measured 1757 seconds at zero real lag). Guarding only the suspended case does not make a commit-delta trigger safe, it converts a silent failure into a noisy one - and the loud half is the likelier to ship, because it appears the first time anyone points it at a database nobody is writing to, whereas the silent half needs someone to suspend something. Evidence only; no behavior change. - **AG fixture evidence: the frozen suspended-row surface, and a lag correction** ([#1707]) - [#1702] and [#1703] both cite `tools/ag-fixture/VALIDATION.md` for how a suspended row behaves, and the file did not yet contain the measurements behind several of those claims. This lands them, with one **correction to my own earlier entry**: [#1704] said `secondary_lag_seconds` latched late (`0` at +15s, then climbing), which implied a short bounded window. On a group with **no write load** it can never latch at all - reproduced independently here, `0` at every sample across a full 60-second suspension while already `NOT SYNCHRONIZING` with the last hardened log **~29 minutes** stale. One earlier idle run did latch at +30s and these did not, so the timing is not dependable in either direction and nothing should be built on it. Stated plainly in the file: **a lag threshold alone cannot detect suspended data movement on a quiet group**; the dedicated suspended-state alert owns that case. Also landed, with sample tables: all four `*_time` columns freeze at their last pre-suspension instant (so a cross-replica commit-time delta stops growing exactly when replication stops); `redo_queue_size` freezes rather than growing; `est_redo_completion_time_min` holds a small static reassuring value throughout a suspension (0.0144 min under load, 0 when idle) because both its queue and rate are frozen, making a suspended replica look *healthier* than a working one; `last_received_time` read NULL in every sample on both runs; and `last_commit_time` is not a heartbeat - measured **1757 seconds behind wall clock on a `SYNCHRONIZED`, non-suspended, zero-lag replica** simply because the database was quiet, so `now - last_commit_time` is not a lag measure and grows without bound on a healthy idle replica. Evidence only; no behavior change. - **AG fixture evidence: what `secondary_lag_seconds` actually measures on a suspended row** ([#1704]) - the collector doc block and the suspended-row alert rule both now cite `tools/ag-fixture/VALIDATION.md` for the claim that the column reports staleness rather than time since suspension; this lands the measurements behind it. Two parties reproduced the accrual on the fixture but from different starting points - 0 under write load, ~3993 on an idle group - and re-testing the idle case explicitly reconciles them: while movement is ACTIVE it reads 0 no matter how long the group has been idle (measured 0 with 373 seconds since the last hardening), and once SUSPENDED it latches onto roughly `now - last_hardened_time` (measured ~450 against 424) and climbs from there. Same behavior, different bases. Also recorded: it does **not** latch the moment movement stops - at +15s suspended it still read `0` while already `NOT SYNCHRONIZING` - which is what makes "a suspended row may raise an alarm but may never clear one" the only safe rule, and the reason the magnitude is staleness rather than volume at risk (`log_send_queue_size` would be the volume measure, and it is NULL while suspended). That late-latch turned out to understate the problem; see [#1707] for the quiet-group case where it never latches at all. Evidence only; no behavior change. - **Darling Web: an "AG Health" seed notebook** ([#1699]) - the fifth Custom Views v2 seed template, built on the Availability Group measures from [#1688] and [#1695]. Five panels in diagnostic order: secondary lag over time grouped by replica; a **dual-axis** line putting log send rate against redo rate on their own axes (the one panel that answers "which side is the bottleneck" - send outpacing redo means the secondary is receiving faster than it can replay, and failover time is growing); a **stacked** redo-queue series by database; a **stat tile** for the worst estimated redo drain; and a top-10 bar of send-queue backlog by database. The prose between panels carries the two things that make the numbers readable rather than merely present: point the view at the PRIMARY (a secondary only ever sees its own replica in `sys.dm_hadr_*`, so a secondary-scoped view is a one-row self-view), and a blank drain estimate means there is no drain rate - idle, caught up, or suspended - not that it drains instantly. Also adds the two template helpers the existing seeds never needed, `overlay` support on the time-series builder and a scalar `statPanel`, so this is the first seed exercising dual-axis, stacked and stat modes end to end. **The templates drift-guard got a real hole closed along the way**: it hand-mirrors each template's panels for validation, but nothing checked the mirror COVERED every template, so a sixth template added without a mirror entry would have gone silently unvalidated until it 400'd in someone's browser. It now reads the template keys out of `notebook.js` and requires the mirror to match exactly, verified by renaming a key and watching it go red. @@ -1681,6 +1682,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 [#1699]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/1699 [#1704]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/1704 [#1707]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/1707 +[#1709]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/1709 [#1702]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/1702 [#1697]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/1697 [#1700]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/1700 diff --git a/tools/ag-fixture/VALIDATION.md b/tools/ag-fixture/VALIDATION.md index 8d39483c6..fc222bd95 100644 --- a/tools/ag-fixture/VALIDATION.md +++ b/tools/ag-fixture/VALIDATION.md @@ -373,6 +373,26 @@ time. last_commit_time` is not a lag measure: on a quiet, entirely healthy replica it grows without bound, and anything alerting on it will page about an idle database. +### Commit-time deltas fail in *both* directions, which is why they are not a lag measure + +The two halves above are separated by a section, so the combined conclusion is easy to miss - and +the combination is the whole argument: + +| Condition | What `now - last_commit_time` does | Failure | +| --- | --- | --- | +| Replica SUSPENDED | `last_commit_time` freezes, so the delta stops growing exactly when replication stops | **Silent** - understates at the moment it is worst | +| Replica healthy but database QUIET | Nothing commits, so the delta grows without bound (measured 1757s at zero real lag) | **Loud** - pages about an idle database | + +Guarding only the suspended case therefore does not make a commit-delta trigger safe; it converts +a silent failure into a noisy one. And the loud half is the more likely to actually ship, because +it appears the first time anyone points the thing at a database nobody is writing to, whereas the +silent half only shows up if you happen to suspend something. + +The conclusion is not "route commit deltas through the suspended-row gate" but **do not derive lag +from commit times at all**. `secondary_lag_seconds` is the lag measure, with the caveats in finding +3; the `*_time` columns are for showing an operator *when* something last happened, not for judging +whether it is late. + Source: [sys.dm_hadr_database_replica_states (Transact-SQL)](https://learn.microsoft.com/en-us/sql/relational-databases/system-dynamic-management-views/sys-dm-hadr-database-replica-states-transact-sql) ## 7. Resource footprint (measured)