Avoid overwriting during backup to support object lock enabled buckets

[kaovilai]

Describe the problem/challenge you have

Object Lock uses a write-once-read-many (WORM) model to store objects.

Synonyms: object retention, object lock, data immutibility

This model breaks velero backup after the introduction of async operations as documented in https://velero.io/docs/v1.12/backup-reference/#limitation under

Cannot support backup data immutability

Starting from 1.11, Velero’s backups may not work as expected when the target object storage has some kind of an “immutability” option configured. These options are known by different names (see links below for some examples). The main reason is that Velero first saves the state of a backup as Finalizing and then checks whether there are any async operations in progress. If there are, it needs to wait for all of them to be finished before moving the backup state to Complete. If there are no async operations, the state is moved to Complete right away. In either case, Velero needs to modify the metadata in object storage and that will not be possible if some kind of immutability is configured on the object storage.

Even with versions prior to 1.11, there was no explicit support in Velero to work with object storage that has “immutability” configuration. As a result, you may see some problems even though backups seem to work (e.g. versions objects not being deleted when backup is deleted).

Note that backups may still work in some cases depending on specific providers and configurations.

For AWS S3 service, backups work because S3’s object lock only applies to versioned buckets, and the object data can still be updated as the new version. But when backups are deleted, old versions of the objects will not be deleted.
Azure Storage Blob supports both versioned-level immutability and container-level immutability. For the versioned-level scenario, data immutability can still work in Velero, but the container-level cannot.
GCP Cloud storage policy only supports bucket-level immutability, so there is no way to make it work in the GCP environment.

Minio is known to break

https://min.io/docs/minio/linux/administration/object-management/object-retention.html
https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock.html
https://docs.hitachivantara.com/r/en-us/content-platform/9.6.x/mk-99arc024/managing-namespaces/configuring-a-namespace/allowing-object-overwrites

It was previously addressed in #1422 so there is some "regression".

Describe the solution you'd like

The main reason is that Velero first saves the state of a backup as Finalizing and then checks whether there are any async operations in progress.

Why can't we just write once on Completed? Can this be opt-in?

Anything else you would like to add:

Environment:

• Velero version (use velero version):
• Kubernetes version (use kubectl version):
• Kubernetes installer & version:
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• 👍 for "The project would be better with this feature added"
• 👎 for "This feature will not enhance the project in a meaningful way"

tangent issue: #8466