Clean up GatewayService

[DaveCTurner]

GatewayService implements logic for delaying state recovery in the cluster until enough data nodes have joined and/or some timeout has occurred. The implementation is pretty old and doesn't make use of much modern infrastructure for doing this kind of thing, making it pretty hard to follow and to improve (see e.g. #86588). We should clean it up and simplify it.

[elasticsearchmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[ywangd]

The implementation is pretty old and doesn't make use of much modern infrastructure for doing this kind of thing

@DaveCTurner Could you plese share more details or examples about the "much modern infrastructure"? I noticed the code has a comment about submitting cluster state update in batch. But it does not feel like related for this issue. The other thing I know is ClusterStateObserver. Is it related here? Though other than ClusterStateObserver#waitForState, it is not very new code?

[DaveCTurner]

Basically I find this class very hard to follow for such a simple feature - if we were to write this functionality today we wouldn't do it like this. I expect we'd use a SubscribableListener for the timeout and one-shot semantics and make things sensitive to the master term rather than having the resetRecoveredFlags() thing.