Correct the commit-delta guidance: it fails loud, not silent (#991) - #1708
Merged
Conversation
#1703 lumped commit-delta triggers in with drain-time triggers and told a future implementer both "fail SILENT rather than loud". For drain-time that is right. For commit deltas it is wrong, and wrong in the direction that matters: it would send someone to guard the suspended case and ship a trigger that pages about every idle database. ag-fixture-builder measured the healthy half while reproducing the never-latch result. On a quiet but PERFECTLY HEALTHY secondary - SYNCHRONIZED, is_suspended = 0, secondary_lag_seconds = 0 - last_commit_time sat 1757 seconds behind wall clock purely because nothing had committed. ag-collector-builder measured the same thing at ~7 minutes on a separate run. It grows without bound on an idle database. So now - last_commit_time fails BOTH ways: it stops growing on a suspended row (the silent half #1703 described) and grows without bound on a quiet healthy one (the loud half, and the one more likely to actually ship, because it shows up immediately in testing on any database nobody is writing to). Guarding only the suspended direction is not enough, so the guidance is no longer "route it through the gate" but "do not derive lag from commit times at all". secondary_lag_seconds is the lag measure; the commit times are for showing an operator WHEN something last happened, not for judging whether it is late. No logic change - this evaluator has never read those columns. It is the doc block a commit-delta trigger would be written into, which is exactly why the guidance in it needs to point the right way. Their reproduction is also the third independent confirmation of the never-latch behavior, and more extreme than mine: zero lag across a full 60-second suspend with the last hardened log ~29 minutes stale, versus ~5 minutes on my run. It does not depend on how stale the group already was; it simply never latched. Darling suite green: 3252 passed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
erikdarlingdata
enabled auto-merge
July 26, 2026 20:01
This was referenced Jul 26, 2026
pull Bot
pushed a commit
to ehtick/PerformanceMonitor
that referenced
this pull request
Jul 29, 2026
erikdarlingdata#1708 corrected the AG lag guidance to "do not derive lag from commit times at all" and cites this file. The file carried both halves of that argument but in separate sections, never drawn together - so a reader who found one half could reasonably conclude the other case was safe to guard against, which is the exact mistake erikdarlingdata#1708 was fixing. Now stated side by side: on a SUSPENDED replica last_commit_time freezes, so now - last_commit_time stops growing exactly when replication stops (silent, understating at the moment it is worst); on a healthy but QUIET database nothing commits, so the same delta grows without bound (loud, measured 1757 seconds at zero real lag). Guarding only the suspended case does not make a commit-delta trigger safe - it converts a silent failure into a noisy one. The loud half is also the likelier to ship, since it appears the first time anyone points it at a database nobody is writing to. Evidence file and CHANGELOG only, no code. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Corrects guidance I wrote in #1703. Doc-only, no logic change.
What was wrong
#1703 lumped commit-delta triggers in with drain-time triggers and told a future implementer that both "fail SILENT rather than loud". For drain-time that is right. For commit deltas it is wrong, and wrong in the direction that matters — it would send someone to guard the suspended case and then ship a trigger that pages about every idle database.
The half I missed
ag-fixture-builder measured it while reproducing the never-latch result. On a quiet but perfectly healthy secondary —
SYNCHRONIZED,is_suspended = 0,secondary_lag_seconds = 0—last_commit_timesat 1757 seconds behind wall clock purely because nothing had committed. ag-collector-builder measured the same thing at ~7 minutes on a separate run. It grows without bound on an idle database.So
now - last_commit_timefails both ways:Guarding only the suspended direction is not enough. The guidance is therefore no longer "route it through the may-fire-never-clear gate" but "do not derive lag from commit times at all".
secondary_lag_secondsis the lag measure; the commit times are for showing an operator when something last happened, not for judging whether it is late.This evaluator has never read those columns. It is simply the doc block a commit-delta trigger would be written into, which is exactly why the guidance in it has to point the right way.
Also on record
ag-fixture-builder's run is the third independent confirmation of the never-latch behavior, and more extreme than mine: zero lag across a full 60-second suspend with the last hardened log ~29 minutes stale, versus ~5 minutes on my run. So it does not depend on how stale the group already was — it simply never latched. They have corrected their own VALIDATION.md, which had described a bounded "+30s window" that someone could have reasoned about.
Testing
Darling suite green: 3252 passed, 0 failed.
🤖 Generated with Claude Code