Move Get Snapshots Serialization to Management Pool

[original-brownbear]

It's in the title. For large index counts generating the response,
both for transport as well as REST layer gets quite expensive and
I've seen a couple of clusters slow-log for verbose=false requests or on the REST layer.
(also it's easy to reproduce in benchmarks when the repo contains 50k indices)
Better generate it off the transport threads.

relates #77466

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[elasticsearchmachine]

Hi @original-brownbear, I've created a changelog YAML for you.

[original-brownbear]

I can see this being a little controversial given that we generate the response on the meta pool if verbose=true. But in those cases I'd argue that we will probably be bound by the IO before the compute such that we never go 100% CPU on all pool threads simultaneously.

[DaveCTurner]

Seems reasonable with the usual concerns about piling more stuff into the MANAGEMENT threadpool, especially on the master, and losing the natural backpressure mechanism of blocking a transport thread. Still, these requests are cancellable so the backlog should eventually reach equilibrium without breaking anything.

We could of course move the serialization of the verbose=true case over to a MANAGEMENT thread too. I think I'd want there to be a limit on the number of in-flight get-snapshots requests before doing that tho.

[original-brownbear]

especially on the master, and losing the natural backpressure mechanism of blocking a transport thread.

Especially on master we don't want that backpressure mechanism IMO :) much worse for its transport threads to slow down.

We could of course move the serialization of the verbose=true case over to a MANAGEMENT thread too.

Not so important in this case I think. The creation of SnapshotInfo from the repository data I think is the slowest part (which isn't happening for ?verbose=true where we read it straight from the repo). Admittedly I haven't benchmarked it in that much detail yet but just looking at the code it's pretty obvious that looking up every index from the repo-data and tim-sorting sorting the indices lists for each snapshot is a little heavy once you reach O(10k) snapshots). The writing of the message to the wire on the transport layer is no big deal I think.
And either way, with this change the REST serialization goes to the MANAGEMENT pool in both cases now, that's the painful one.

This one isn't as bad as cluster stats or /_mappings btw. When benchmarking some crazy stuff like 200 snapshots of 50k indices I'm around a 10s block on master for most requests. You'll probably get hit by response size issues before this action gets so slow that requests are piling up right now. This is mainly about removing the warning which has been logged here and there lately and ruling this out as a source of master instability.

[DaveCTurner]

LGTM

[original-brownbear]

Thanks David!

[elasticsearchmachine]

💚 Backport successful

Status	Branch	Result
✅	8.0
✅	7.17