Stagger background thread spin-down to avoid a thundering herd

[spacejam]

No description provided.

[ghost]

Thanks!

[yoshuawuyts]

10 secs spin down sounds a bit on the high end to me, but admittedly my intuition here is mostly theoretical. I'd be keen to talk more about this at a later time to understand it better (:

[spacejam]

@yoshuawuyts This is an autoscaling system. One of the most common mistakes when building such systems is to release processing resources too quickly, repeatedly incurring latency hiccups during spin-up when load bursts.

That said, this would probably be better if it followed a FILO-based threadpool approach, rather than a FIFO which will keep threads alive as long as they receive any work before their timeout, keeping 10k threads open without all that much actually happening. This would let most of the threads time-out when mostly idle.