Proxied requests query other nodes in parallel #2779
This change makes the
The HTTP requests are being executed on a shared thread pool, which has a maximum size, defined by the
Motivation and Context
For large cluster sizes, performing proxied requests sequentially could lead to large round trip times, which might exceed the defined timeout. This could lead to functionality being unavailable or even an overloaded Graylog server.
How Has This Been Tested?
Node cleanup was disabled and different ranges of dummy node table entries were created, with transport addresses pointing to a local http stub returning dummy metrics responses. Then, a cluster metrics request was sent to the Graylog server and round trip times were measured.
Types of changes
I am not really happy about introducing another thread pool to the system. A small Graylog server already has around 160 threads in several pools. (last time I checked) Most of them are idle, though. I would rather like to have a few configurable thread pools for different purposes than every new or modified subsystem adding their own thread pools.
But I guess that's a larger refactoring and we shouldn't do that right now.
So for the pool sizing I would like to avoid adding 64 threads by default. The HTTP requests in the proxied resources are currently single threaded, so I would actually use a pretty low default like 2 or 4 which already is an improvement for smaller setups. In bigger setups where this is still a problem the pool size needs to be adjusted.
* Call other nodes concurrently for proxied requests. * Injecting ExecutorService in ProxiedResource + configured max pool size * Explaing proxied_requests_max_threads config parameter in sample config. * Making config value consistent, changing default, explaining sizing. (cherry picked from commit 87acd4a)