pg templates: correct the README failover timings and ship the backup warning that missed #478 - #479
Merged
Merged
Conversation
… is a failed run
Fixed IN PLACE - README and lastModified only, so `helm template` output is
byte-identical before and after on both charts (verified).
Testing found that a failed logical backup still leaves a ~20-byte empty gzip in
the bucket under a normal-looking timestamped filename, because the upload is a
pipeline:
pg_dumpall ... | gzip | aws s3 cp - "s3://.../${FILENAME}"
`set -euo pipefail` makes the job exit non-zero, but gzip emits a valid empty
stream on EOF and the upload completes before the shell reacts. Measured: the bad
object gunzips to 0 lines, a good one from the same cluster to 163 lines with
CREATE ROLE statements. Nothing distinguishes them but size.
The real fix belongs in the backup-images repo (dump to a temp file, verify it is
non-empty, then upload) and is being raised there. This is the template-side
mitigation so nobody restores an empty dump in the meantime.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…hange left stale Fixed IN PLACE - README and lastModified only; `helm template` output verified byte-identical on all three charts. Two problems, both introduced when the DCS timeouts were retuned to 45/10/15 and the READMEs were not updated with them: The tables still advertised ttl 60s and retry_timeout 20s, and the patronictl edit-config examples still told users to set 60 / 10 / 20 - which would have moved a cluster AWAY from what the chart now ships. That is the same README-vs-values drift this audit has been closing everywhere else, introduced by the fix for it. The tables also claimed ttl "sets the worst-case failover delay". Measurement does not support that. Abrupt recovery took ~70s at ttl 60 and ~70s at ttl 45 - lowering ttl did not move it. What did move was the graceful case, from ~7s to ~4s, and that came from the HAProxy fall 3 -> 2 change rather than from ttl. Replaced with what was actually measured: a planned failover releases the lock immediately and costs a few seconds, with Patroni's own handover at 185ms and the HAProxy health check being what clients wait on; an abrupt loss takes about a minute, and ttl is the lock-expiry bound rather than a dial for failover speed. Stated as measured rather than derived, because the abrupt case is simulated by SIGSTOPing Patroni's worker (PID 1 is signal-immune from inside the container), PID 1 stays alive, and there was one run per configuration. That may not isolate the ttl term, so the honest claim is that ttl was not the dominant term in this test. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Follow-up to #478, from the in-depth test session you asked for. Everything here is in place — README and
lastModifiedonly — withhelm templateoutput verified byte-identical on all three charts, so no version is bumped.Two corrections, both to problems the #478 work itself created:
1. The empty-object warning never shipped. It was still uncommitted when #478 was squash-merged, so it isn't in
main. Caught by diffing the branch againstmainrather than trusting that "merged" meant everything landed.2. The README timings went stale. Retuning the DCS values to 45/10/15 left the tables advertising
ttl 60s/retry_timeout 20s— and thepatronictl edit-configexamples still told users to set60 / 10 / 20, which would move a cluster away from what ships. That is the exact README-vs-values drift this audit has been closing everywhere else, introduced by the fix for it.The
ttlclaim was not supported by measurementThe tables said
ttl"sets the worst-case failover delay". Measured on a 3-member cluster:Lowering
ttldid not move abrupt recovery. What did move was the graceful case — 6 failed writes over ~7s became 3 over ~4s — and that came from the HAProxyfall 3 → 2change, not fromttl. Patroni's own handover is 185 ms; clients wait on the health check.Replaced with measured behaviour, and stated as measured rather than derived because the method has a real limit: the abrupt case is simulated by
SIGSTOP-ing Patroni's worker (PID 1 is signal-immune from inside the container), PID 1 stays alive, and there was one run per configuration. So the honest claim is thatttlwas not the dominant term in this test.Test session results (against the merged 2.6.0 / 1.2.0 / 1.1.0)
failsafe_modemultiZonereplicas: 1etcd.replicas: 1internal_access: workload-listUnchanged, zeroUpdatedNot covered, stated plainly
postgres-multi-locationhad no in-container verification.cpln workload execinto the GVC it creates is denied by the policy hook;workload getreads are permitted,execis not. So nopatronictl, no runtime confirmation thatfailsafe_modeis active, no credential round-trip. Its functional changes are the same values proven end-to-end on the other two, but this is not the same standard of evidence.backup-images, not this repo. This PR only adds the warning so nobody restores a zero-length dump meanwhile.🤖 Generated with Claude Code