Consider Postgres health during rolling updates

[sdudoladov]

It happens now that a rolling update may terminate a replica that currently undergoes recovery. This
(a) leaves a replica in an undefined state and (b) is somewhat hard to debug.

One can extend the operator to query Patroni API and at least log the PG state when attempting to re-create a pod during the rolling update.

[redbaron]

Is it already safe for the opposite case: shutting down master which has zero up to date replicas without waiting for replicas to catch up?

[Jan-M]

Good point. While we often assume persistent volumes that make us in worst case experience a slightly longer downtime other users may rely on ephemeral storage. Operating any of this without s3 or similar for continuous archiving or EBS semantics is obviously not ideal or dangerous.

[redbaron]

@Jan-M , how EBS helps here? If postgres-operator makes a switch (does it just kill pod or does the switch like patronictl is doing it?) then replica starts new history from the point where it was, right? meaning that master is left with some committed, but not accessible anymore transactions.

[Jan-M]

I was thinking along the lines of an unhealthy replica scenario. Where the replicas would not be promoted. But the master pod gets deleted and then K8S reschedules it and wherever it comes back it will be the pod most ahead in time and it will promote to master again.

I saw this as some danger for users without persistent volumes, where this will not work and end up all data being lost.

Your point is valid though, if within the min promote distance and no data is/was shipped to s3 than a replica may promote and data lost @CyberDem0n ?

[redbaron]

@Jan-M ,

if within the min promote distance

does postgres-operator take min promote distance into account at all?

[Jan-M]

The operator leaves this to Patroni, interacting with the Patroni API to do the switch over.

[FxKu]

fixed with #903