Choose WAL-G backup by start time when cloning and deleting

[simonklb]

Fixes https://github.com/elastisys/compliantkubernetes-postgresql/issues/241

Also fixes the following upstream issues AFAIK (more investigation needed to verify):

• Bump wal-g v3:
  • Bump wal-g to v3 zalando/spilo#986
• Restore issues:
  • WAL-E Clone should not use last_modified for recovery target time evaluation zalando/spilo#576
  • Wrong timestamp taken during base backup recovery zalando/spilo#652
• Clean up issues:
  • Backups not being deleted as expected zalando/spilo#937
  • backup retention logic in /scripts/postgres_backup.sh is broken if basebackup frequency is low zalando/spilo#425
  • Default DAYS_TO_RETAIN value zalando/spilo#879

Does not fix when Spilo relies on wal-g backup-list LATEST:

• Relying on modification time may lead to unexpected results (issues with backup-list, and backup-fetch .. LATEST) wal-g/wal-g#694
• Sort backups by the creation time in generic backup-list command wal-g/wal-g#1108

Breaks cloning from backup that does not include metadata.json (for example backup created with wal-e):

• backup-list --detail fails on missing metadata.json wal-g/wal-g#556

[OlleLarsson]

Thanks! Will take a look early next week!

[OlleLarsson]

LGTM!

[davidumea]

👍

[simonklb]

This fixes the immediate problem of cleaning up backups based on start time instead of modified time: fcb00b1

Note that this is not at all backwards compatible with wal-e, so don't merge this! ⚠️

However, when working on this I found this: wal-g/wal-g#816 which suggests that there is already built-in support for this. Also, using delete retain instead of delete before could also slim down the script even further as it currently tries to calculate how many backups to retain on it's own instead of relying on wal-g.

[simonklb]

Here's a new patch that makes use of wal-g delete retain --after: bb77d67
PTAL!

Diff it with ignore whitespace and it'll look less intimidating. :)

[simonklb]

FWIW the Spilo tests passes with these changes cherry-picked on master and with USE_WALG=true.

However, without USE_WALG=true some tests fail logs similar to this:

2024-06-20 16:08:59,805 INFO: Trying s3://testbucket/spilo/demo/wal/12/ for clone
INFO: 2024/06/20 16:08:59.822165 List backups from storages: [default]
INFO: 2024/06/20 16:08:59.823516 No backups found
2024-06-20 16:08:59,825 INFO: Trying s3://testbucket/spilo/demo/wal/11/ for clone
INFO: 2024/06/20 16:08:59.840049 List backups from storages: [default]
ERROR: 2024/06/20 16:08:59.841804 object 'base_000000010000000000000005_00000208/metadata.json' not found in storage
2024-06-20 16:08:59,843 ERROR: Clone failed

This is expected because if the backups are created with wal-e the metadata.json file is never created, so if we want to try to get this merged upstream we'll probably have to work around this as well or query if wal-e support is planned to be dropped in the near future.

Note that this is should only be an issue if you are trying to migrate from wal-e to wal-g. Using wal-e exclusively should still work.

[OlleLarsson]

...
This is expected because if the backups are created with wal-e the metadata.json file is never created, so if we want to try to get this merged upstream we'll probably have to work around this as well or query if wal-e support is planned to be dropped in the near future.

Note that this is should only be an issue if you are trying to migrate from wal-e to wal-g. Using wal-e exclusively should still work.

I think it makes sense to ask if they are planning on removing wal-e support.

[OlleLarsson]

Ugh, running into some issues building the image from this branch, seem to fail on building wal-g 🤔

[100%] Built target brotli
+ bash link_libsodium.sh

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
The command '/bin/sh -c bash /builddeps/dependencies.sh' returned a non-zero code: 2

[simonklb]

Ugh, running into some issues building the image from this branch, seem to fail on building wal-g 🤔

[100%] Built target brotli
+ bash link_libsodium.sh

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
The command '/bin/sh -c bash /builddeps/dependencies.sh' returned a non-zero code: 2

wal-g/wal-g#1738

[simonklb]

Blocked by: wal-g/wal-g#1738

Temp fix to allow for testing while we wait for the fix to be released: 8416ba1