AG fixture: land the secondary_lag_seconds evidence the doc blocks cite (#991) - #1704
Merged
Merged
Conversation
An independent reproduction of finding 3 on this fixture measured the same accrual but starting near 3993 rather than 0, on an idle AG. Re-testing the idle case explicitly explains the difference, and the explanation is more useful than the original finding. While data movement is ACTIVE, secondary_lag_seconds reads 0 no matter how long the AG has been idle - measured 0 with 373 seconds since the last hardening. Once SUSPENDED it latches onto roughly how stale the secondary's last hardened log is (now - last_hardened_time, measured ~450 against 424) and climbs from there. Under write load last_hardened_time is always near-now, so it starts at ~0 and looks like time-since-suspension; on an idle AG it starts at however long since the last write. Same behavior, different starting point. Two things that matter for thresholding, both now recorded: - On an idle AG a lag rule can fire instantly on suspension with a large number reflecting idleness, not data at risk. The magnitude is staleness of the last hardening; log_send_queue_size would be the volume measure, and that is NULL while suspended. - The column does not update the moment movement stops. At +15s suspended it still read 0 while already NOT SYNCHRONIZING, latching only at +30s. A rule that reads 0 from a suspended row as "caught up" can clear an alarm in that window. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
erikdarlingdata
enabled auto-merge
July 26, 2026 19:37
# Conflicts: # CHANGELOG.md
4 tasks
pull Bot
pushed a commit
to ehtick/PerformanceMonitor
that referenced
this pull request
Jul 29, 2026
…ag claim erikdarlingdata#1702 and erikdarlingdata#1703 both cite tools/ag-fixture/VALIDATION.md for how a suspended row behaves, and the file did not contain the measurements behind several of those claims. This lands them. It also corrects something I got wrong in erikdarlingdata#1704. I wrote that secondary_lag_seconds "only latched at the +30s sample", which implied a short bounded window. ag-alerts-builder measured it never latching across a whole 60s suspension on a quiet group, and I reproduced that independently here: 0 at every sample through a full 60-second suspension, already NOT SYNCHRONIZING, with the last hardened log ~29 minutes stale. One earlier idle run latched at +30s and these did not, so the timing is not dependable in either direction. The file now says so, and states the consequence plainly: a lag threshold alone cannot detect suspended data movement on a quiet group. Also added, with sample tables from a loaded run and an idle one: - All four *_time columns freeze at their last pre-suspension instant, so a cross-replica commit-time delta stops growing exactly when replication stops - the opposite direction from secondary_lag_seconds. - redo_queue_size freezes rather than growing. - est_redo_completion_time_min is the sharpest edge: queue over rate with both frozen holds a small static reassuring number for the whole suspension (0.0144 min loaded, 0 idle), so a suspended replica looks healthier than a working one. - last_received_time read NULL in every sample on both runs. - last_commit_time is not a heartbeat - measured 1757 seconds behind wall clock on a SYNCHRONIZED, non-suspended, zero-lag replica, purely because the database was quiet. Loaded-run table is ag-collector-builder's; the idle run and the never-latch reproduction are mine. Evidence file and CHANGELOG only, no code. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Small follow-up to #1689. Evidence only, no behavior change - one file plus a CHANGELOG entry.
Why
Both the collector doc block on
devand the suspended-row rule in flight now citetools/ag-fixture/VALIDATION.mdfor the claim thatsecondary_lag_secondsreports staleness rather than time since suspension, and that a suspended row can still read0for a sample or two. Those measurements were taken but never landed - #1689 was merged while I was still measuring, so the two commits after the merge were orphaned on the branch. This lands the half that is still needed.What it adds
@ag-alerts-builder reproduced the lag accrual independently on the fixture but measured it starting near 3993 where I measured 0. Their
redo_queue_sizewas 0 throughout - an idle AG, where mine was under write load. Re-testing the idle case explicitly reconciles both:While movement is active the column reads 0 no matter how long the group has been idle. Once suspended it latches onto roughly
now - last_hardened_timeand climbs. Under loadlast_hardened_timeis always near-now, so it starts at ~0 and looks like time-since-suspension; on an idle group it starts at however long since the last write. Same behavior, different bases - neither measurement was wrong.Two consequences, both now recorded next to the numbers:
log_send_queue_sizewould be the volume measure and it is NULL while suspended.0while alreadyNOT SYNCHRONIZING. That window is exactly why "a suspended row may raise an alarm but may never clear one" is the only safe rule - a rule reading that0as caught-up would clear an alarm mid-fault.What I deliberately dropped
I had also corrected the collector's doc comment on the old branch. That commit is dropped as redundant -
feature/991-ag-suspension-semanticsalready carried the same refinement intoAgDatabaseReplicaStatesCollector.csand went further (the four*_timecolumns freezing, the never-cross-check rule). Pushing mine would have collided with two in-flight branches editing that file. Nothing here touches any collector, so this cannot conflict with them.Test plan
dev; single file, no collector or app code touched.feature/991-ag-lag-semanticsorfeature/991-ag-suspension-semantics- neither touchestools/ag-fixture/.Fixture is still up and healthy: ag1
localhost,14331(primary), ag2localhost,14332.🤖 Generated with Claude Code