Support standby replication from GS (GCS)

[jamesmcdonald]

This allows standby clusters to be created from a GCS bucket by specifying either gs_wal_path with a full manual path, or cluster and optionally uid and version for similar behaviour to cloning. I've updated the documentation with the details. I have tested it against a number of GCS WAL-G backups and it seems to work nicely. I haven't tested the cluster-based replication from S3 as I don't have any data in S3 to try it out with.

It uses the main GCPCredentials from the operator config rather than requiring a standby-specific setting which I think should generally work. If different credentials are needed for pushing WALs then they can be changed at the same time as the standby is promoted.

I created this before I realised there was an existing PR! Let me know if it's useful. This is going to be an essential feature for us so I'm happy to amend it as needed.

[FxKu]

Thanks @jamesmcdonald. Why would you need the version. Yes, since Spilo 13, it's part of the WAL path, but then you would have it s3/gs_wal_path, no? I also do not see that it is appended somewhere. Only an env variable is set that Spilo doesn't know about. Aside from that it's looking fine I think.

[jamesmcdonald]

Thanks for the review! As I understand, PGVERSION gets added to the bucket path here: https://github.com/zalando/spilo/blob/master/postgres-appliance/scripts/configure_spilo.py#L763 after it's set from STANDBY_PGVERSION by https://github.com/zalando/spilo/blob/master/postgres-appliance/scripts/configure_spilo.py#L1027.

[FxKu]

No, PGVERSION should already be available for every spilo pod depending on the version chosen in the Postgres cluster manifest. You don't have to define it again. We also don't do this on cloning.

[jamesmcdonald]

Right, I see what you mean. That certainly makes sense given that you can only create a standby from the same version. Unfortunately the main PGVERSION isn't picked up by the current logic in Spilo, so if STANDBY_PGVERSION isn't set then the WALG_GS_PREFIX in /var/run/etc/wal-e.d/env-standby-* ends up just ending in /wal with no version.

I guess doing this properly would require additional logic in Spilo to either:

• Guess the version, like https://github.com/zalando/spilo/blob/master/postgres-appliance/bootstrap/clone_with_wale.py#L131-L134 does for cloning, or
• Use PGVERSION to configure the standby environment context

I've updated my branch to remove the version setting and demonstrate this behaviour; there's another branch https://github.com/jamesmcdonald/postgres-operator/tree/gs-standby-version with the previous commit.

Do you have a suggestion for the best way to proceed?

[jamesmcdonald]

I've removed the ability to select a path via cluster/uid/version entirely because it doesn't work reliably. It's pretty straightforward to use the s3/gs_wal_path options.

[FxKu]

Suggested change

	if description.S3WalPath != "" {
	if description.S3WalPath == "" && description.GSWalPath == "" {
	return nil
	}
	if description.S3WalPath != "" {

We should still consider the case when either one of the WAL paths is empty and return nil to have the same behavior.

[jamesmcdonald]

Good point, fixed!

[FxKu]

this can go, I think. Maybe a left over from an earlier version of this PR

[jamesmcdonald]

Cleaned these up too.

[FxKu]

@jamesmcdonald can you have a look at my two comments? I think after that we can merge this PR :)

[FxKu]

👍

[Jan-M]

👍