Prevent Alfred Telemetry Prometheus endpoint threads from stacking up

[thijslemmens]

A client had an issue where fetching some metrics was getting very slow, slower then the scraping frequency of Prometheus. After a while the http threads where being exausted.

Possible solution: Wrap the the fetching of metrics in a separate thread that gets killed when taking too long, returning an error message.

[todorinskiz]

Use our own threadpool with a configurable keepAliveTime?https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/ThreadPoolExecutor.html#ThreadPoolExecutor-int-int-long-java.util.concurrent.TimeUnit-java.util.concurrent.BlockingQueue-java.util.concurrent.ThreadFactory-java.util.concurrent.RejectedExecutionHandler-

Also worth reading with regards to the thread pool:
http://www.bigsoft.co.uk/blog/2009/11/27/rules-of-a-threadpoolexecutor-pool-size

[thijslemmens]

Or wrapping in a Future with a timeout:
https://stackoverflow.com/questions/2275443/how-to-timeout-a-thread

We just have to check if that timeout would also stop the query.

[kerkhofsd]

My main concern is: how are we going to define this timeout?

This needs to match exactly with the Prometheus poll interval? If it's less, Prometheus requests might be aborted. If it's more, threads might still stack up?

[thijslemmens]

My main concern is: how are we going to define this timeout?

This needs to match exactly with the Prometheus poll interval? If it's less, Prometheus requests might be aborted. If it's more, threads might still stack up?

You are right. A timeout could be an "extra". Controlling concurrency is probably the more structural fix. We can use a lock:
https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/locks/ReentrantLock.html

There is a tryLock() method that returns false immediately if another thread holds the lock. In that case Telemetry can return an error code. This way you can limit Telemetry to only have one active thread at the same time. Would 'one' be acceptable?