Skip to content

AG fixture: land the secondary_lag_seconds evidence the doc blocks cite (#991) - #1704

Merged
erikdarlingdata merged 3 commits into
devfrom
feature/991-ag-fixture-lag-evidence
Jul 26, 2026
Merged

AG fixture: land the secondary_lag_seconds evidence the doc blocks cite (#991)#1704
erikdarlingdata merged 3 commits into
devfrom
feature/991-ag-fixture-lag-evidence

Conversation

@erikdarlingdata

Copy link
Copy Markdown
Owner

Small follow-up to #1689. Evidence only, no behavior change - one file plus a CHANGELOG entry.

Why

Both the collector doc block on dev and the suspended-row rule in flight now cite tools/ag-fixture/VALIDATION.md for the claim that secondary_lag_seconds reports staleness rather than time since suspension, and that a suspended row can still read 0 for a sample or two. Those measurements were taken but never landed - #1689 was merged while I was still measuring, so the two commits after the merge were orphaned on the branch. This lands the half that is still needed.

What it adds

@ag-alerts-builder reproduced the lag accrual independently on the fixture but measured it starting near 3993 where I measured 0. Their redo_queue_size was 0 throughout - an idle AG, where mine was under write load. Re-testing the idle case explicitly reconciles both:

idle, NOT suspended       lag=0    (373s since last hardening - still 0)
idle, NOT suspended +20s  lag=0
idle, suspended +15s      lag=0    NOT SYNCHRONIZING
idle, suspended +30s      lag=450  (secs_since_hardened 424)
idle, suspended +45s      lag=465  (secs_since_hardened 440)
after resume              lag=0

While movement is active the column reads 0 no matter how long the group has been idle. Once suspended it latches onto roughly now - last_hardened_time and climbs. Under load last_hardened_time is always near-now, so it starts at ~0 and looks like time-since-suspension; on an idle group it starts at however long since the last write. Same behavior, different bases - neither measurement was wrong.

Two consequences, both now recorded next to the numbers:

  • On an idle AG a lag rule fires instantly on suspension with a large value reflecting idleness, not data at risk. The magnitude is staleness; log_send_queue_size would be the volume measure and it is NULL while suspended.
  • It does not latch when movement stops. At +15s suspended it still read 0 while already NOT SYNCHRONIZING. That window is exactly why "a suspended row may raise an alarm but may never clear one" is the only safe rule - a rule reading that 0 as caught-up would clear an alarm mid-fault.

What I deliberately dropped

I had also corrected the collector's doc comment on the old branch. That commit is dropped as redundant - feature/991-ag-suspension-semantics already carried the same refinement into AgDatabaseReplicaStatesCollector.cs and went further (the four *_time columns freezing, the never-cross-check rule). Pushing mine would have collided with two in-flight branches editing that file. Nothing here touches any collector, so this cannot conflict with them.

Test plan

  • Idle-case measurements taken on the running fixture, and they reconcile the two prior datasets rather than contradicting either.
  • Cherry-picked clean onto current dev; single file, no collector or app code touched.
  • No overlap with feature/991-ag-lag-semantics or feature/991-ag-suspension-semantics - neither touches tools/ag-fixture/.

Fixture is still up and healthy: ag1 localhost,14331 (primary), ag2 localhost,14332.

🤖 Generated with Claude Code

erikdarlingdata and others added 2 commits July 26, 2026 15:27
An independent reproduction of finding 3 on this fixture measured the same
accrual but starting near 3993 rather than 0, on an idle AG. Re-testing the
idle case explicitly explains the difference, and the explanation is more
useful than the original finding.

While data movement is ACTIVE, secondary_lag_seconds reads 0 no matter how
long the AG has been idle - measured 0 with 373 seconds since the last
hardening. Once SUSPENDED it latches onto roughly how stale the secondary's
last hardened log is (now - last_hardened_time, measured ~450 against 424) and
climbs from there. Under write load last_hardened_time is always near-now, so
it starts at ~0 and looks like time-since-suspension; on an idle AG it starts
at however long since the last write. Same behavior, different starting point.

Two things that matter for thresholding, both now recorded:

- On an idle AG a lag rule can fire instantly on suspension with a large
  number reflecting idleness, not data at risk. The magnitude is staleness of
  the last hardening; log_send_queue_size would be the volume measure, and
  that is NULL while suspended.
- The column does not update the moment movement stops. At +15s suspended it
  still read 0 while already NOT SYNCHRONIZING, latching only at +30s. A rule
  that reads 0 from a suspended row as "caught up" can clear an alarm in that
  window.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@erikdarlingdata
erikdarlingdata merged commit 35dc49a into dev Jul 26, 2026
4 checks passed
@erikdarlingdata
erikdarlingdata deleted the feature/991-ag-fixture-lag-evidence branch July 26, 2026 20:19
pull Bot pushed a commit to ehtick/PerformanceMonitor that referenced this pull request Jul 29, 2026
…ag claim

erikdarlingdata#1702 and erikdarlingdata#1703 both cite tools/ag-fixture/VALIDATION.md for how a suspended
row behaves, and the file did not contain the measurements behind several of
those claims. This lands them.

It also corrects something I got wrong in erikdarlingdata#1704. I wrote that
secondary_lag_seconds "only latched at the +30s sample", which implied a short
bounded window. ag-alerts-builder measured it never latching across a whole
60s suspension on a quiet group, and I reproduced that independently here: 0
at every sample through a full 60-second suspension, already NOT
SYNCHRONIZING, with the last hardened log ~29 minutes stale. One earlier idle
run latched at +30s and these did not, so the timing is not dependable in
either direction. The file now says so, and states the consequence plainly: a
lag threshold alone cannot detect suspended data movement on a quiet group.

Also added, with sample tables from a loaded run and an idle one:

- All four *_time columns freeze at their last pre-suspension instant, so a
  cross-replica commit-time delta stops growing exactly when replication
  stops - the opposite direction from secondary_lag_seconds.
- redo_queue_size freezes rather than growing.
- est_redo_completion_time_min is the sharpest edge: queue over rate with both
  frozen holds a small static reassuring number for the whole suspension
  (0.0144 min loaded, 0 idle), so a suspended replica looks healthier than a
  working one.
- last_received_time read NULL in every sample on both runs.
- last_commit_time is not a heartbeat - measured 1757 seconds behind wall
  clock on a SYNCHRONIZED, non-suspended, zero-lag replica, purely because the
  database was quiet.

Loaded-run table is ag-collector-builder's; the idle run and the never-latch
reproduction are mine. Evidence file and CHANGELOG only, no code.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant