Detach a Postgres cluster from the operator temporarily

[Jan-M]

We were facing a situation where we needed to edit the statefulset of one cluster manually. This does not work while the operator is running and syncing objects.

We solved this by scaling operator to 0, but this is not ideal.

Suggest we look into the manifest to add something like:

sync: false
detach: true
maintenance: true

Or similar.

[Jan-M]

This change may need to be observed out of order, not sure how this could be done. But ideally it does not first apply all kind of pending changes on this cluster but stops sync immediately.

In first round though this could always be achieved with manual operator restart.

[valer-cara]

👍 Similar issue here. I have a 3-node cluster, and the first one is down because I got some EBS errors on an AWS M5 machine which rendered the volume read-only (dmesg here).

I wanted to unmount the volume and do a fsck on it, however to do that i'll need to stop the operator, delete the sts with --cascade=false to avoid pod re-creating, delete the affected pod & manually start debugging the volume. Finally restarting the operator would recreate the STS & the downed pod.

A maintenance feature per cluster would be most useful.

---

PS: for issues that don't require unmounting volumes one can simply use patronictl -c ./postgres.yaml pause to stop patroni temporarily and do whatever's needed. more here

[valer-cara]

So maybe there are multiple levels of maintenance that can be expressed in the cluster manifest?

maintenance:
  mode: paused

• pause: where patroni is paused on the cluster
• detach: where the operator doesn't monitor k8s resources anylonger
• full: all the above?

Not sure if this is the best approach or naming, just trying to underline the types of "detachment" out there.

[sdudoladov]

one special case to watch out is the incomplete rolling update

1. The operator starts the rolling update of the cluster
2. the first replica gets restarted but for whatever reason cannot join the cluster again
3. this blocks the rolling upgrade

[FxKu]

With #802 merged this could be realized via acid.zalan.do/conrtoller annotations. So far, it's manual process. Closing for now.