Some obscure k8s-backup solutions break init on single-Node setup

[Ornias1993]

Some obscure backup/migration solutions, such as TrueNAS SCALE their "single node cluster backups", edit/recreate the PV/PVC/CR's on recovery.

CNPG correctly finds the PV and starts, however it still seems to think it needs init and starts the init jobs (vaultwarden-cnpg-main-1-initdb) which throws the following error:
PGData already exists, can't overwrite

In general this makes me think that there might be more resiliency needed for single node scenario's and initdb detecting existing cluster data.

[sxd]

Hi @Ornias1993

this looks like something that can be fixed using the hibernation procedure, now, since this is more like an infrastructure thing I'm moving this into a discussion until we have a clear issue.

Regards!

[Ornias1993]

This is not an infrastructure issue afaik.
Not all platforms can automatically set labels/annotations on node shutdown. Actually: most cannot.

I doubt hibernate is the fix here, because it has nothing to do with the fact the init job thinking it's not-yet initialised when it actually has a PVC mounted with existing data. That does not change with hibernation afaik.

[Ornias1993]

In hindsight, I think this is very related to:
#1286

So i'll close this in favor of #1286

Whereas one is more about volumesnapshotting in a k8s-native way, both issues (k8s snapshots and host snapshots) seem to be hitting this precise same bug on the init job. So are related enough to keep in one thread.