100% CPU in selector on Windows

[amshf]

There is a Java web-app that is running by Jetty 9.4.3.v20170317
After some minutes/hours of working, the web-app starts to use 100% CPU without any specific reason. Even when it is idle.

After profiling and getting thread-dumps, it seems that Jetty is the source of high CPU consumption.

So what is wrong with Jetty? is this a bug? (Please pay attention to the second image)

Below I list some evidences after profiling (using VisualVM) when CPU consumption reaches to 100%:

• Running threads (CLICK to enlarge images):
  

• CPU Sampler:
  

• CPU Sampler-per thread:
  

• Memory Sampler:
  

• Process status (using processExplorer):
  

• Process performance (using processExplorer):
  

[joakime]

What does the thread dump for that thread look like over time? (Does its stacktrace change?)
What version of Java? (Are you running the latest JVM?)
What OS? (there's been sporadic reports of NIO issues on Windows, esp with various home firewall software or network optimization software installed on dev/test machines)

[joakime]

More questions...
What are the clients connecting to your server? (be specific)
What is the timeout configuration on your server?
What is your server attempting to do?

[amshf]

Let me describe the test scenarios first:
I've tested the app on:
-two OS,
-on Eclipse (by Jetty) OR run it by Jetty itself,
-by disabling all of web-app libraries (like disruptor, netty, executerServices, connectionPools, TCPClient and Server,...),
-on several Jetty versions,
-several times, the same results.

---

What does the thread dump for that thread look like over time? (Does its stacktrace change?)

Some of the top running threads yes and some of them no.

• the first stackTrace(today)
  AllThreads.txt

• the second stackTrace(today, after 50 mins from the first one)
  AllThreads3-after-50-minutes.txt

• the image of top running threads:
  

What version of Java? (Are you running the latest JVM?)

On two JDK versions: (jdk1.8.0 AND jdk1.8.0_121)

What OS? (there's been sporadic reports of NIO issues on Windows, esp with various home firewall

On Windows 7 and Windows 10
With and without windows firewall.

software or network optimization software installed on dev/test machines)

Nothing. just an UTM for all the network.

What are the clients connecting to your server? (be specific)

this is a web-app with panels for end-users. no clients were connected at the test environment. the web app was idle.

What is the timeout configuration on your server?

what type of timeouts do you mean?
http timeout is 10,000 and also 30,000 (default) has been tested.
<Set name="idleTimeout"><Property name="jetty.http.idleTimeout" deprecated="http.timeout" default="10000"/></Set>

What is your server attempting to do?

this is a panel for traders. some market info is gathered and pushed to UI, some executor-services are running and MongoDB+SQLServer are in back-end. there is some queues (Disruptor) and messages get in and out by Netty. (all of this functionalities are disabled in test scenarios)

[joakime]

Please try 9.4.4-SNAPSHOT located at the maven snapshot repository
https://oss.sonatype.org/content/repositories/jetty-snapshots/

We've been making improvements to the threading as a result of Issue #1440 , and these have proven useful for the Windows specific load scenarios as well.

[amshf]

Tried 9.4.4-SNAPSHOT (jetty-distribution-9.4.4-20170405.091100-9), but unfortunately got the same results.

[ChaitanyaKiranPVN]

Hi joakime,

I am new to the Jetty server .Currently i am deploying my application using jetty . please find the below server spec
CentOS release 5.6 (Final) & Centos 7
8GB Ram,
jdk-8u112-linux-x64
jetty-distribution-9.4.1.v20170120

i am also facing the same issue in Linux environment with Jetty 9.4.1 which Consumes 100% CPU once started the jetty server.

please find the attached thread dump file
threaddump-1493364860829.txt

Thread details

CPU details

Please let me know how to resolve this issue asap.

Thanks
Chaitanya

[gregw]

Can you please retry with Jetty-9.4.4 as we believe several issues that could have caused your problem have been fixed since 9.4.1

[joakime]

This is not reproducible in latest Jetty 9.4.x series.

[amshf]

I don't see it again by using 9.4.6.v20170531
I think the problem is fixed in this version.
Thanks.

[buko]

I think this bug hit me hard when running Eclipse. Is there a way to upgrade the version of Jetty that an Eclipse instance uses?

[jmcc0nn3ll]

sorry, you will have to ask the eclipse folks, we have nothing to do with how it is used inside of eclipse

[ihor-zhakun]

Hi

I've got the same problem, when very simple project (from the comment of related issue) after 2 hrs (+/- few minutes) got 100% CPU usage
hazelcast/hazelcast#7943 (comment)

Env: Windows 10, jdk build 1.8.0_171-b11, tried also with jdk v71, v161
also different versions of hazelcast: 3.6.2, 3.8.8, 3.9.4, 3.10.
Problem always is reproducing with 100% chance after 2 hrs of idle.

In debug there is next info, if compare data before and after CPU overhead:
sun.nio.ch.WindowsSelectorImpl.SubSelector#poll0
before - readFds array is fulled with 0
after - readFds[0] == 1, value of readFds[1] == sun.nio.ch.WindowsSelectorImpl#wakeupSourceFd
also method poll0 is executing instantly, without any timeout and thread is in endless loop without block or wait.

Any help on the problem? Or any advise what to debug else or try to change?