Skip to content

Correct the commit-delta guidance: it fails loud, not silent (#991) - #1708

Merged
erikdarlingdata merged 2 commits into
devfrom
feature/991-ag-commit-delta-note
Jul 26, 2026
Merged

Correct the commit-delta guidance: it fails loud, not silent (#991)#1708
erikdarlingdata merged 2 commits into
devfrom
feature/991-ag-commit-delta-note

Conversation

@erikdarlingdata

Copy link
Copy Markdown
Owner

Corrects guidance I wrote in #1703. Doc-only, no logic change.

What was wrong

#1703 lumped commit-delta triggers in with drain-time triggers and told a future implementer that both "fail SILENT rather than loud". For drain-time that is right. For commit deltas it is wrong, and wrong in the direction that matters — it would send someone to guard the suspended case and then ship a trigger that pages about every idle database.

The half I missed

ag-fixture-builder measured it while reproducing the never-latch result. On a quiet but perfectly healthy secondary — SYNCHRONIZED, is_suspended = 0, secondary_lag_seconds = 0last_commit_time sat 1757 seconds behind wall clock purely because nothing had committed. ag-collector-builder measured the same thing at ~7 minutes on a separate run. It grows without bound on an idle database.

So now - last_commit_time fails both ways:

  • silent on a suspended row — the delta stops growing exactly when replication stops (what Say what secondary_lag_seconds actually measures (#991) #1703 described)
  • loud on a quiet healthy one — grows without bound, and this is the half more likely to actually ship, because it shows up immediately in testing against any database nobody is writing to

Guarding only the suspended direction is not enough. The guidance is therefore no longer "route it through the may-fire-never-clear gate" but "do not derive lag from commit times at all". secondary_lag_seconds is the lag measure; the commit times are for showing an operator when something last happened, not for judging whether it is late.

This evaluator has never read those columns. It is simply the doc block a commit-delta trigger would be written into, which is exactly why the guidance in it has to point the right way.

Also on record

ag-fixture-builder's run is the third independent confirmation of the never-latch behavior, and more extreme than mine: zero lag across a full 60-second suspend with the last hardened log ~29 minutes stale, versus ~5 minutes on my run. So it does not depend on how stale the group already was — it simply never latched. They have corrected their own VALIDATION.md, which had described a bounded "+30s window" that someone could have reasoned about.

Testing

Darling suite green: 3252 passed, 0 failed.

🤖 Generated with Claude Code

erikdarlingdata and others added 2 commits July 26, 2026 16:00
#1703 lumped commit-delta triggers in with drain-time triggers and told a future
implementer both "fail SILENT rather than loud". For drain-time that is right.
For commit deltas it is wrong, and wrong in the direction that matters: it would
send someone to guard the suspended case and ship a trigger that pages about
every idle database.

ag-fixture-builder measured the healthy half while reproducing the never-latch
result. On a quiet but PERFECTLY HEALTHY secondary - SYNCHRONIZED,
is_suspended = 0, secondary_lag_seconds = 0 - last_commit_time sat 1757 seconds
behind wall clock purely because nothing had committed. ag-collector-builder
measured the same thing at ~7 minutes on a separate run. It grows without bound
on an idle database.

So now - last_commit_time fails BOTH ways: it stops growing on a suspended row
(the silent half #1703 described) and grows without bound on a quiet healthy one
(the loud half, and the one more likely to actually ship, because it shows up
immediately in testing on any database nobody is writing to). Guarding only the
suspended direction is not enough, so the guidance is no longer "route it
through the gate" but "do not derive lag from commit times at all".
secondary_lag_seconds is the lag measure; the commit times are for showing an
operator WHEN something last happened, not for judging whether it is late.

No logic change - this evaluator has never read those columns. It is the doc
block a commit-delta trigger would be written into, which is exactly why the
guidance in it needs to point the right way.

Their reproduction is also the third independent confirmation of the never-latch
behavior, and more extreme than mine: zero lag across a full 60-second suspend
with the last hardened log ~29 minutes stale, versus ~5 minutes on my run. It
does not depend on how stale the group already was; it simply never latched.

Darling suite green: 3252 passed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@erikdarlingdata
erikdarlingdata merged commit 97ed4f2 into dev Jul 26, 2026
4 checks passed
@erikdarlingdata
erikdarlingdata deleted the feature/991-ag-commit-delta-note branch July 26, 2026 20:16
pull Bot pushed a commit to ehtick/PerformanceMonitor that referenced this pull request Jul 29, 2026
erikdarlingdata#1708 corrected the AG lag guidance to "do not derive lag from commit times
at all" and cites this file. The file carried both halves of that argument but
in separate sections, never drawn together - so a reader who found one half
could reasonably conclude the other case was safe to guard against, which is
the exact mistake erikdarlingdata#1708 was fixing.

Now stated side by side: on a SUSPENDED replica last_commit_time freezes, so
now - last_commit_time stops growing exactly when replication stops (silent,
understating at the moment it is worst); on a healthy but QUIET database
nothing commits, so the same delta grows without bound (loud, measured 1757
seconds at zero real lag). Guarding only the suspended case does not make a
commit-delta trigger safe - it converts a silent failure into a noisy one. The
loud half is also the likelier to ship, since it appears the first time anyone
points it at a database nobody is writing to.

Evidence file and CHANGELOG only, no code.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant