Fork CCS remote-cluster responses

[DaveCTurner]

Today during CCS we process responses from remote clusters on the
transport worker thread, but that's not the right place for this work.
This commit moves it to the SEARCH_COORDINATION pool.

Closes #97997

[elasticsearchmachine]

Pinging @elastic/es-search (Team:Search)

[elasticsearchmachine]

Hi @DaveCTurner, I've created a changelog YAML for you.

[quux00]

I did a little ad-hoc testing of this change one of my CCS branches in progress and it seemed the same as before based on my limited testing of it (e.g., I'm currently dealing with a strange async-search CCS race condition and it happens on both SEARCH_COORDINATION threadpool and the DIRECT/transport pool).

I don't know a lot about ES threadpool details. Are there any limitations or considerations I should be aware of that this SEARCH_COORDINATION threadpool has compared to the "DIRECT" (transport) thread pool?

[DaveCTurner]

Are there any limitations or considerations I should be aware of

We have to be very careful about what we run on the transport pool. We still use it far too much, with cluster-destroying consequences sometimes, although these days it's not as bad as it used to be. A lot of the places we use it are because historically this has been the default choice, but it's a really bad default choice: we really should default to another pool unless there's very good reasons to keep work on the transport pool. CCS responses in particular can be too large to reasonably process on the transport worker, and CCS has enough latency that we can afford a few µs of forking overhead, and I believe if DIRECT hadn't been the default up until #98131 then we would have chosen a different pool years ago.

In contrast the SEARCH_COORDINATION pool intended for the bits of searches that are kinda lightweight (IO-free) but not light enough to run directly on the transport pool. Today that mostly means the can-match phase, but processing CCS responses also fits into this category IMO. It has a bounded queue with default length of 1000; the bound does not apply to responses, but if the queue was full of response handling then we would start rejecting requests. One slight concern is that it defaults to having ½#CPUs threads, with a max of 5, whereas there are #CPUs transport threads. But then again there's no work-sharing between transport threads so you can hit bottlenecks at any scale there. I can see a few more things that I think make sense to move to SEARCH_COORDINATION, and with enough things happening here I could see an argument for expanding it to having #CPUs threads.

[quux00]

LGTM