Jetty fails with IllegalStateException on hosts with many cores

[mhilbush]

On hosts with many cores (somewhere around 16 or more), Jetty throws an IllegalStateException when starting up. As the webclient config parameters are buried in the code, there's no obvious way to figure out how to work around the problem (other than finding my post on the forum).

16:25:43.381 [ERROR] [et.http.internal.WebClientFactoryImpl] - Could not start Jetty http client
java.lang.IllegalStateException: Insufficient configured threads: required=13 < max=10 for QueuedThreadPool[common]@75f3f957{STARTED,5<=5<=10,i=5,q=0}[ReservedThreadExecutor@16f08868{s=0/1,p=0}]
        at org.eclipse.jetty.util.thread.ThreadPoolBudget.check(ThreadPoolBudget.java:149) ~[?:?]
        at org.eclipse.jetty.util.thread.ThreadPoolBudget.leaseTo(ThreadPoolBudget.java:130) ~[?:?]
        at org.eclipse.jetty.util.thread.ThreadPoolBudget.leaseFrom(ThreadPoolBudget.java:175) ~[?:?]
        at org.eclipse.jetty.io.SelectorManager.doStart(SelectorManager.java:251) ~[?:?]
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[?:?]
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138) ~[?:?]
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) ~[?:?]
        at org.eclipse.jetty.client.AbstractConnectorHttpClientTransport.doStart(AbstractConnectorHttpClientTransport.java:64) ~[?:?]
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[?:?]
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138) ~[?:?]
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) ~[?:?]
        at org.eclipse.jetty.client.HttpClient.doStart(HttpClient.java:241) ~[?:?]
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[?:?]
        at org.eclipse.smarthome.io.net.http.internal.WebClientFactoryImpl$2.run(WebClientFactoryImpl.java:258) ~[?:?]
        at org.eclipse.smarthome.io.net.http.internal.WebClientFactoryImpl$2.run(WebClientFactoryImpl.java:1) ~[?:?]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
        at org.eclipse.smarthome.io.net.http.internal.WebClientFactoryImpl.createHttpClientInternal(WebClientFactoryImpl.java:244) ~[?:?]
        at org.eclipse.smarthome.io.net.http.internal.WebClientFactoryImpl.access$7(WebClientFactoryImpl.java:242) ~[?:?]
        at org.eclipse.smarthome.io.net.http.internal.WebClientFactoryImpl$1.run(WebClientFactoryImpl.java:218) ~[?:?]
        at org.eclipse.smarthome.io.net.http.internal.WebClientFactoryImpl$1.run(WebClientFactoryImpl.java:1) ~[?:?]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:?]
        at org.eclipse.smarthome.io.net.http.internal.WebClientFactoryImpl.initialize(WebClientFactoryImpl.java:209) ~[?:?]
        at org.eclipse.smarthome.io.net.http.internal.WebClientFactoryImpl.getCommonHttpClient(WebClientFactoryImpl.java:160) ~[?:?]
        at org.eclipse.smarthome.io.net.http.HttpUtil.executeUrlAndGetReponse(HttpUtil.java:194) ~[?:?]
        at org.eclipse.smarthome.io.net.http.HttpUtil.downloadData(HttpUtil.java:438) ~[?:?]
        at org.eclipse.smarthome.io.net.http.HttpUtil.downloadImage(HttpUtil.java:401) ~[?:?]
        at org.eclipse.smarthome.io.net.http.HttpUtil.downloadImage(HttpUtil.java:372) ~[?:?]
        at org.eclipse.smarthome.io.net.http.HttpUtil.downloadImage(HttpUtil.java:358) ~[?:?]
        at org.openhab.binding.squeezebox.internal.handler.SqueezeBoxPlayerHandler.lambda$0(SqueezeBoxPlayerHandler.java:419) ~[?:?]```

[kaikreuzer]

Insufficient configured threads: required=13

Do you happen to know how Jetty calculates the minimal number based on the cores?
We could then set it accordingly in https://github.com/eclipse/smarthome/blob/master/bundles/io/org.eclipse.smarthome.io.net/src/main/java/org/eclipse/smarthome/io/net/http/internal/WebClientFactoryImpl.java#L173.

[mhilbush]

I tracked it as far as here, but didn't get to the actual calculation.
https://github.com/eclipse/jetty.project/blob/c7fc674846577bbdf33585638a42ea46e0a00768/jetty-util/src/main/java/org/eclipse/jetty/util/thread/ThreadPoolBudget.java#L148

[kaikreuzer]

What is weird is that the exception says max=10, while we set max to 40 and only min to 10 for the shared client.

I checked the code and it seems we are not mixing up min&max anywhere. So I have no idea, why it thinks that max should be limited to 10...?

[mhilbush]

Yeah that had me scratching my head too. 😕

I just did a test on my 6-core box.

With these settings

org.eclipse.smarthome.webclient:minThreadsShared=2
org.eclipse.smarthome.webclient:maxThreadsShared=3
org.eclipse.smarthome.webclient:minThreadsCustom=2
org.eclipse.smarthome.webclient:maxThreadsCustom=4

I got this exception. So for some reason could it be using the maxThreadsCustom setting?

2018-11-27 09:23:47.347 [DEBUG] [rnal.handler.SqueezeBoxPlayerHandler] - Trying to download the content of URL http://192.168.1.31:9000/plugins/Pandora/html/images/icon.png
2018-11-27 09:23:47.349 [DEBUG] [t.http.internal.WebClientFactoryImpl] - creating http client for consumer common, endpoint null
2018-11-27 09:23:47.350 [DEBUG] [t.http.internal.WebClientFactoryImpl] - Setting up SSLContext for org.eclipse.smarthome.io.net.http.internal.ExtensibleTrustManager@57db6a9a
2018-11-27 09:23:47.355 [ERROR] [t.http.internal.WebClientFactoryImpl] - Could not start Jetty http client
java.lang.IllegalStateException: Insufficient configured threads: required=4 < max=4 for QueuedThreadPool[common]@7e414b1a{STARTED,2<=2<=4,i=2,q=0}[ReservedThreadExecutor@30ead769{s=0/1,p=0}]
        at org.eclipse.jetty.util.thread.ThreadPoolBudget.check(ThreadPoolBudget.java:149) ~[?:?]
        at org.eclipse.jetty.util.thread.ThreadPoolBudget.leaseTo(ThreadPoolBudget.java:130) ~[?:?]

[mhilbush]

And, I just changed maxThreadsCustom to 5 and it started up fine.

[kaikreuzer]

The bug seems to be here: It invariably creates and sets a new ThreadPool with maxThreadsCustom threads for the common client :-/

[kaikreuzer]

The threadpool that is created here does not ever seem to be used :-(

[mhilbush]

Ugh. I can't believe I didn't spot that when reading through the code.

[kaikreuzer]

#6588 shoud fix it.

[mhilbush]

Thanks!

So, now that it's using the correct max of 40, the likelihood of throwing that error becomes greatly reduced, except if the host has a lot of cores.

[kaikreuzer]

Right and if you are running such a massively parallel machine, you are still free to add an individual configuration for the pool size.

[mhilbush]

Yep. While the likelihood of someone running into this is now very low, I'll update my forum post to be more explicit about which parameter needs to be changed.