Migrate S3's `CompareAndExchangeOperation` to use conditional writes

[DaveCTurner]

AWS S3 now has support for strongly-consistent conditional writes from which one may build a safe compare-and-exchange operation. Moreover, AWS have now withdrawn their earlier claims about the linearizability of multipart upload operations on which today's implementation is based. Thus for safety we must migrate to using conditional writes for linearizable register operations in S3-backed repositories.

However, customers running third-party S3-compatible storage solutions may not yet have upgraded their storage to one which supports this conditional write feature, but which does already have a properly linearizable multipart upload subsystem. We generally have to be quite strict about requiring full compatibility with AWS S3, but in recognition of the time it takes to upgrade a production storage implementation I propose we support both compare-and-exchange mechanisms for a while. Users may indicate that their storage does not support conditional writes using the setting introduced in #137185.

We must also consider how to ensure that a running cluster can safely switch from using linearizable multipart uploads over to using conditional writes during a rolling upgrade.

[elasticsearchmachine]

Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)

[0xShubhamSolanki]

Hi @DaveCTurner,
I've gone through the codebase and understand that we need to:

1.Add a new conditional-write-based CaS implementation using S3's If-Match header
2.Fall back to the existing multipart-based CaS when repositories.s3.allow_multipart_cas is true
3.Ensure safe rolling upgrades by relying on the cluster-wide setting

I'm ready to implement this with proper tests and backward compatibility.
Could you please assign this issue to me?

[DaveCTurner]

Hi @0xShubhamSolanki, thanks for your interest but no we cannot assign issues to external contributors. Moreover this one is pretty tricky and is not a good choice for an external contributor.