AG fixture: land the suspended-row evidence, and correct the lag latch claim (#991)

[erikdarlingdata]

Evidence file and CHANGELOG only, no code. Two files.

#1702 and #1703 both cite tools/ag-fixture/VALIDATION.md for how a suspended row behaves, and the file did not contain the measurements behind several of those claims. This lands them - and corrects one of mine.

Correction to #1704

I wrote that secondary_lag_seconds "only latched at the +30s sample", which implied a short bounded window. That was too generous. @ag-alerts-builder measured it never latching at all across a full 60-second suspension on a quiet group, and I reproduced it independently rather than take it on report:

elapsed  is_suspended  sync_state         lag  secs_since_hardened
healthy  0             SYNCHRONIZED       0    1757
0s       1             NOT SYNCHRONIZING  0    1757
+15s     1             NOT SYNCHRONIZING  0    1772
+30s     1             NOT SYNCHRONIZING  0    1788
+45s     1             NOT SYNCHRONIZING  0    1803
+60s     1             NOT SYNCHRONIZING  0    1818
resume   0             SYNCHRONIZED       0    16

Zero at every sample, already NOT SYNCHRONIZING, last hardened log ~29 minutes stale. One earlier idle run latched at +30s and these did not, which is the actual lesson: the timing is not dependable in either direction and nothing should be built on it.

Stated plainly in the file now, because the intuitive assumption is the opposite:

A lag threshold alone cannot detect suspended data movement on a quiet group. It can read 0 for the entire outage.

The dedicated suspended-state alert owns that case. This is what makes #1700's "a sub-threshold reading on a suspended row is never caught up" load-bearing for a far more common scenario than the inverted-docs one it was written for.

Also landed (new section 6b)

With sample tables from a loaded run and an idle one:

• All four *_time columns freeze at their last pre-suspension instant, so a cross-replica commit-time delta stops growing exactly when replication stops - understating the problem at the moment it is worst, the opposite direction from secondary_lag_seconds.
• redo_queue_size freezes (21896 KB flat) rather than growing.
• est_redo_completion_time_min is the sharpest edge. Queue over rate with both frozen holds a small static reassuring value for the whole suspension - 0.0144 min under load, 0 when idle - so a suspended replica looks healthier than a working one. est_send_drain_time_min at least reads NULL.
• last_received_time read NULL in every sample, healthy and suspended alike, on both runs. Optional rather than expected.
• last_commit_time is not a heartbeat. Measured 1757 seconds behind wall clock on a SYNCHRONIZED, non-suspended, zero-lag replica, purely because the database was quiet. So now - last_commit_time is not a lag measure - on a healthy idle replica it grows without bound, and anything alerting on it pages about an idle database. This one bites on healthy replicas, not just suspended ones.

The summary table at the top is updated so it no longer contradicts any of this.

Attribution

The loaded-run table is @ag-collector-builder's, supplied as raw data; the idle run, the never-latch reproduction, and the last_commit_time baseline are mine. @ag-alerts-builder found the never-latch case first.

Test plan

• Never-latch reproduced independently on the fixture rather than accepted on report - it contradicted my own published claim, so it needed verifying.
• *_time freeze, last_received_time NULL, and the last_commit_time gap all confirmed in my own idle run as well.
• CHANGELOG: entry plus link-ref, no duplicate [#1702]/[#1703] refs, no conflict markers.
• Diff against dev is exactly CHANGELOG.md + tools/ag-fixture/VALIDATION.md - no collector or app code, so it cannot conflict with anything in flight.

Fixture left as found: both replicas SYNCHRONIZED, is_suspended = 0, healthy.

🤖 Generated with Claude Code