Add a configurable cooldown period between SLM operations

[dakrone]

For some of our repository types (particularly S3), there can sometimes be issues due to eventual visibility of files that have been uploaded.

To handle this from SLM, we should add a configurable "cooldown" period in between operations (snapshot and delete).

[elasticmachine]

Pinging @elastic/es-core-features (:Core/Features/ILM)

[jasontedor]

This doesn't feel right to me, that our solution to "eventual consistency" problems is to wait a little bit, while we will still have no guarantees that how long we wait will make what we're waiting on visible. I think @original-brownbear has been working so that we aren't as susceptible to eventual consistency issues, so I really want to challenge the necessity of this. Since he's spent far more time thinking about this, I would solicit his input here.

[original-brownbear]

@dakrone @jasontedor yea, I talked about this with @ywelsch this morning and I gotta admit I'm a little at fault for suggesting we move ahead with this.

The situation is this:

As it stands now we are still succeptible to S3 eventual consistency issue unfortunately. The impact of it has been reduced step by step over the last year though. #46250 will reduce the impact to a single operation during snapshotting only and a follow up should be able to fix the eventual consistency issue for good.
My assumption when suggesting we move the wait from operations into ES/SLM was that:

1. It would be very simple to add that wait to SLM
2. We'd be able to remove it transparently once it has become unnecessary

we will still have no guarantees that how long we wait will make what we're waiting on visible

That's true. Unfortunately though, currently waiting is what we do in operations to work around these issues. In practice the wait we're using turns out to be safe across a very large sample size.

=> I think what this comes down to is:
If SLM will schedule operations (create/delete) on the repo in rapid succession that could be a problem with S3 repositories. Do we want to fix this problem for users or can we wait another minor for the eventual consistency fixes to land?

[jasontedor]

It would be my preference that we wait another minor release for the eventual consistency fixes to land than we hack around this, and building infrastructure that will otherwise be unused (#47522) for this.

[dakrone]

I agree I'd rather wait until this was handled at the snapshot/restore side rather than with infra in SLM. It is actually not very simple to add to SLM, the dedicated #47522 threadpool is the "simplest" way I could think of to address it, I'd be more than happy not to have to add it.

[dakrone]

Given that we have agreed (I think?) to address this through consistency fixes in snapshot/delete itself, I'm going to close this as well.