Add wait_if_ongoing for refresh API increasing refresh reliability

[luyuncheng]

Description

Problem Statement

When we have many frequently update and refresh request for many indices (maybe every write 4 docs, refresh 1, and Write 1000 docs per seconds every shards, i know refresh is a resource-intensive api as docs says: ).

it shows that refresh queue would blocked hundreds of thousands of queued requests even in 3 nodes with 16G heap, 10 primary shards, 20 replications shards, and 400G storage, ES Version 8.4.3, 96 Core CPU

I think this because

1. REFRESH thread pool type is ThreadPoolType.SCALING
2. TransportShardRefreshAction is extends of TransportReplicationAction which forceExecution is default true in replication(like TransportReplicationAction.java#L200 )
3. REFRESH API would call InternalEngine refresh with block = true (like InternalEngine.java#L1795 ) and in hot threads shows block in acquire lock
   

So a refresh request would expands to indices * shards * replications (in our test case is 30), with blocked executions

Proposal

May be we can add a wait_if_ongoing parameter in refresh api like flush api. which can make refresh requests with nonblocking. just calling InternalEngine#maybeRefresh. when it can not acquire a lock, it must be a in-flight refresh task is running

PR #91578

[nik9000]

Indexing has a already has a wait_for_refresh option which is pretty similar. Is that useful for you?

[elasticsearchmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[luyuncheng]

Indexing has a already has a wait_for_refresh option which is pretty similar. Is that useful for you?

@nik9000 thanks for the replying.
i think this is a good way to wait for doc until they refreshed in few write per seconds. But when refresh_inteval = '10s+' AND many writes per seconds, these requests which are waiting for refresh occupied many memory.
i try to use this parameter, as http client is a pipeline model in http1.1, client must wait for the response, so the client and coordianate node GC frequency goes up, and write performance drops down.

meanwhile, i think this parameter can make the refresh api more robust that would not pending 200,000 requests at only 30 shards.

[ywangd]

I am closing this as a duplicate of #87936.