EPollArrayWrapper.epollWait 100% CPU Usage #327

blucas · 2012-05-16T13:02:07Z

Hi,

I believe I have an issue similar to #302 but on Linux (Ubuntu 10.04) with JDK (1.6.0u30) and JDK(1.7.0u4) using Netty-4.0.0 (Revision: 52a7d28)

The app is proxying connections to backend systems. The proxy has a pool of channels that it can use to send requests to the backend systems. If the pool is low on channels, new channels are spawned and put into the pool so that requests sent to the proxy can be serviced. The pools get populated on app startup, so that is why it doesn't take long at all for the CPU to spike through the roof (22 seconds into the app lifecycle).

The test box has two CPUs, the output from 'top' is below:

PID  USER   PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
8220 root   20   0 2281m 741m  10m R 50.2 18.7  0:22.57 java                                                                             
8218 root   20   0 2281m 741m  10m R 49.9 18.7  0:22.65 java                                                                             
8219 root   20   0 2281m 741m  10m R 49.2 18.7  0:22.86 java                                                                             
8221 root   20   0 2281m 741m  10m R 49.2 18.7  0:22.20 java

Thread Dump for the four NioClient based Worker Threads that are chewing up all the CPU.

"backend-worker-pool-7-thread-1" prio=10 tid=0x00007f5918015800 nid=0x201a runnable [0x00007f5924ba3000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    - locked <0x000000008be93580> (a sun.nio.ch.Util$2)
    - locked <0x000000008be93570> (a java.util.Collections$UnmodifiableSet)
    - locked <0x000000008be92548> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    at io.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:55)
    at io.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:261)
    at io.netty.channel.socket.nio.NioWorker.run(NioWorker.java:37)
    at io.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:43)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)   Locked ownable synchronizers:    - <0x000000008be00748> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"backend-worker-pool-7-thread-2" prio=10 tid=0x00007f5918012000 nid=0x201b runnable [0x00007f5924b82000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    - locked <0x000000008be94a28> (a sun.nio.ch.Util$2)
    - locked <0x000000008be94a18> (a java.util.Collections$UnmodifiableSet)
    - locked <0x000000008be90648> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    at io.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:55)
    at io.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:261)
    at io.netty.channel.socket.nio.NioWorker.run(NioWorker.java:37)
    at io.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:43)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)   Locked ownable synchronizers:    - <0x000000008be904c8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"backend-worker-pool-7-thread-3" prio=10 tid=0x00007f5918007800 nid=0x201c runnable [0x00007f5924b61000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    - locked <0x000000008be952e0> (a sun.nio.ch.Util$2)
    - locked <0x000000008be952d0> (a java.util.Collections$UnmodifiableSet)
    - locked <0x000000008be8f858> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    at io.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:55)
    at io.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:261)
    at io.netty.channel.socket.nio.NioWorker.run(NioWorker.java:37)
    at io.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:43)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)   Locked ownable synchronizers:    - <0x000000008be8f618> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"backend-worker-pool-7-thread-4" prio=10 tid=0x00007f5918019000 nid=0x201d runnable [0x00007f5924b40000]
   java.lang.Thread.State: RUNNABLE
    at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    - locked <0x000000008be003f8> (a sun.nio.ch.Util$2)
    - locked <0x000000008be003e8> (a java.util.Collections$UnmodifiableSet)
    - locked <0x000000008be00408> (a sun.nio.ch.EPollSelectorImpl)
    at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    at io.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:55)
    at io.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:261)
    at io.netty.channel.socket.nio.NioWorker.run(NioWorker.java:37)
    at io.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:43)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)   Locked ownable synchronizers:    - <0x000000008be004e0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

The text was updated successfully, but these errors were encountered:

blucas · 2012-05-16T17:31:31Z

I've done some googling and it looks like this could be a JDK bug.
Take a look at the thread dump supplied on 13-MAY-2009. It is exactly the same as mine! Although they say the bug has been fixed, it obviously hasn't. Could you please investigate the workaround supplied in the link? They mention cancelling the selector and invoking a selectNow() method. I have NO clue where to do that myself to test if it actually works, so if someone could change it in the repo, I'm more than happy to give it a test.

In fact I've actually found a post by @trustin on this very subject. But it looks like no changes were made back then.

blucas · 2012-05-18T10:39:06Z

@trustin @normanmaurer Any thoughts on this?

Eirenliel · 2012-06-14T14:52:49Z

Please, update. Merged on new server wit j6u33 and now experiencing this bug. Will try to downgrade jre...

Eirenliel · 2012-06-14T16:10:07Z

It is happens on Netty 3.5.0 Final, 3.2.7 Final and 3.4.7 j6u33 and may be others.

twinforces · 2012-07-09T17:19:34Z

Yeah, I see this too with UltraESB.

I think its a problem with anything that uses the Non-blocking IO library.

incubos · 2012-07-26T17:01:48Z

I confirm this bug on Netty 3.3.0 Final (from Akka 2.0.2) & j7u4.

vikiitd · 2012-08-02T10:45:11Z

Hi,

We are seeing this issue in our production environment also under heavy concurrent load. Are there any updates/fix for this issue. On some forums there are discussions related to this issue being resolved by upgrading jdk, etc. I am not seeing this issue with netty.3.2.4.Final.jar but it is happening consistently with netty3.4.4.Final.jar Does anyone know if there is a work-around for this issue besides downgrading back to 3.2.4 netty jar version.

Thanks in advance!

vikiitd · 2012-08-08T06:40:46Z

I am observing this issue with Netty-3.2.6.Final.jar also, although less frequently. Is it possible that this issue got introduced before 3.3 or it is not a netty issue at all?

trustin · 2012-08-09T08:52:48Z

Is there any reliable way to reproduce this problem? Then it would be much easier for me to fix this issue (probably it's more correct to say 'apply workaround').

vrajesh1989 · 2012-08-12T15:09:20Z

Hi. This issue is occurring in Netty-3.2.4.Final.jar also. There is no reliable way to reproduce this issue, it is not happening consistently.

normanmaurer · 2012-08-23T19:11:17Z

Let me see if I can fix this ...

…% cpu load. This is just a workaround but helps to recover. See #327

normanmaurer · 2012-08-25T19:58:35Z

@trustin I think this is because of an epoll bug which still is not fixed or was fixed and is now present again.

Please review the following workaround:
#565

So basically we do the following here:

check if Selector.select(timeout) returns 0 and complete before the timeout
if so we increment a counter
if the counter == 10 we create a new selector and register all the Channels from the old one to the new one and close the old Selector. After that we reset the counter.
if the counter < 10 we just process
if Selector.select(timeout) returns > 0 or complete after the timeout we reset the counter

The fix is kind of the same as jetty,grizzly and mina do :) WDYT ?

vikiitd · 2012-08-27T05:14:06Z

it will be great if we can have the workaround provided by @normanmaurer . The jdk bug seems to be hanging around for sometime now, not sure if it is getting fixed anytime soon.

normanmaurer · 2012-08-27T05:28:49Z

@vikiitd I have lost the hope to get nio bugs fixed in core java a long time ago :(

…and #327

normanmaurer · 2012-08-29T06:47:17Z

Workaround for the epoll(..) bug was commited.

…ual wakup or because of a closed channel. See #327

…jboss.netty.epollBugWorkaround=true. It will be enabled by default later if it has proven to be stable. See #327

zaffargachal · 2017-02-22T13:16:35Z

@endlesstian - Did you find the solution of your problem. I am facing the same problem with Netty 4.1.0.CR7, Jdk 1.8. and almost 10k thread in tomcat. Tomcat became irresponsive

senthilec566 · 2017-07-04T15:05:55Z

Looks like in recent version netty also has same issue !

OS :
Ubuntu 14.04.5 LTS (GNU/Linux 3.13.0-121-generic x86_64)

Java:
java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)

CPU Usage:
ps -o '%cpu,%mem' -p 2023
%CPU %MEM
287 7.9

Netty Version:
4.1.12.Final

jstack

"epollEventLoopGroup-3-8" #20 prio=10 os_prio=0 tid=0x00007f10ac016000 nid=0x811 runnable [0x00007f10b731a000]
java.lang.Thread.State: RUNNABLE
at io.netty.channel.epoll.Native.epollWait0(Native Method)
at io.netty.channel.epoll.Native.epollWait(Native.java:117)
at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:226)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:250)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
at java.lang.Thread.run(Thread.java:748)

"epollEventLoopGroup-3-7" #19 prio=10 os_prio=0 tid=0x00007f10ac014000 nid=0x810 runnable [0x00007f10b741b000]
java.lang.Thread.State: RUNNABLE
at io.netty.channel.epoll.Native.epollWait0(Native Method)
at io.netty.channel.epoll.Native.epollWait(Native.java:117)
at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:226)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:250)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
at java.lang.Thread.run(Thread.java:748)

"epollEventLoopGroup-3-6" #18 prio=10 os_prio=0 tid=0x00007f10ac012800 nid=0x80f runnable [0x00007f10b751c000]
java.lang.Thread.State: RUNNABLE
at io.netty.channel.epoll.Native.epollWait0(Native Method)
at io.netty.channel.epoll.Native.epollWait(Native.java:117)
at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:226)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:250)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
at java.lang.Thread.run(Thread.java:748)

"epollEventLoopGroup-3-5" #17 prio=10 os_prio=0 tid=0x00007f10ac010800 nid=0x80e runnable [0x00007f10b761d000]
java.lang.Thread.State: RUNNABLE
at io.netty.channel.epoll.Native.epollWait0(Native Method)
at io.netty.channel.epoll.Native.epollWait(Native.java:117)
at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:226)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:250)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
at java.lang.Thread.run(Thread.java:748)

"epollEventLoopGroup-3-4" #16 prio=10 os_prio=0 tid=0x00007f10ac00e800 nid=0x80d runnable [0x00007f10b771e000]
java.lang.Thread.State: RUNNABLE
at io.netty.channel.epoll.Native.epollWait0(Native Method)
at io.netty.channel.epoll.Native.epollWait(Native.java:117)
at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:226)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:250)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
at java.lang.Thread.run(Thread.java:748)

senthilec566 · 2017-07-04T15:46:28Z

@normanmaurer - is this issue still exist in 4.X ??

doom369 · 2017-07-04T16:02:17Z

@senthilec566 are you sure you have 4.1.12.Final? These line numbers don't seem to be from 4.1.12.Final.

senthilec566 · 2017-07-04T16:46:48Z

@doom369 - Yes , doubled checked the jar ..

cat pom.properties
#Created by Apache Maven 3.3.9
version=4.1.12.Final
groupId=io.netty
artifactId=netty-all

ps -o '%cpu,%mem' -p 6696
%CPU %MEM
449 8.2

epollEventLoopGroup-3-13" #25 prio=10 os_prio=0 tid=0x00007fcc3804e800 nid=0x1a6a runnable [0x00007fcc42dcb000]
java.lang.Thread.State: RUNNABLE
at io.netty.channel.epoll.Native.epollWait0(Native Method)
at io.netty.channel.epoll.Native.epollWait(Native.java:111)
at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:230)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:254)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
at java.lang.Thread.run(Thread.java:748)

"epollEventLoopGroup-3-12" #24 prio=10 os_prio=0 tid=0x00007fcc3804c800 nid=0x1a69 runnable [0x00007fcc42ecc000]
java.lang.Thread.State: RUNNABLE
at io.netty.channel.epoll.Native.epollWait0(Native Method)
at io.netty.channel.epoll.Native.epollWait(Native.java:111)
at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:230)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:254)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
at java.lang.Thread.run(Thread.java:748)

"epollEventLoopGroup-3-11" #23 prio=10 os_prio=0 tid=0x00007fcc3804a800 nid=0x1a68 runnable [0x00007fcc42fcd000]
java.lang.Thread.State: RUNNABLE
at io.netty.channel.unix.FileDescriptor.writeAddress(Native Method)
at io.netty.channel.unix.FileDescriptor.writeAddress(FileDescriptor.java:135)
at io.netty.channel.epoll.AbstractEpollChannel.doWriteBytes(AbstractEpollChannel.java:290)
at io.netty.channel.epoll.AbstractEpollStreamChannel.writeBytes(AbstractEpollStreamChannel.java:260)
at io.netty.channel.epoll.AbstractEpollStreamChannel.doWriteSingle(AbstractEpollStreamChannel.java:472)
at io.netty.channel.epoll.AbstractEpollStreamChannel.doWrite(AbstractEpollStreamChannel.java:457)

normanmaurer · 2017-07-04T16:48:32Z

Can you please attach a profiler snapshot ?

…

On 4. Jul 2017, at 18:46, Senthilkumar K ***@***.***> wrote: @doom369 <https://github.com/doom369> - Yes , doubled checked the jar .. cat pom.properties #Created by Apache Maven 3.3.9 version=4.1.12.Final groupId=io.netty artifactId=netty-all ps -o '%cpu,%mem' -p 6696 %CPU %MEM 449 8.2 epollEventLoopGroup-3-13" #25 <#25> prio=10 os_prio=0 tid=0x00007fcc3804e800 nid=0x1a6a runnable [0x00007fcc42dcb000] java.lang.Thread.State: RUNNABLE at io.netty.channel.epoll.Native.epollWait0(Native Method) at io.netty.channel.epoll.Native.epollWait(Native.java:111) at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:230) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:254) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) at java.lang.Thread.run(Thread.java:748) "epollEventLoopGroup-3-12" #24 <#24> prio=10 os_prio=0 tid=0x00007fcc3804c800 nid=0x1a69 runnable [0x00007fcc42ecc000] java.lang.Thread.State: RUNNABLE at io.netty.channel.epoll.Native.epollWait0(Native Method) at io.netty.channel.epoll.Native.epollWait(Native.java:111) at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:230) at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:254) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) at java.lang.Thread.run(Thread.java:748) "epollEventLoopGroup-3-11" #23 <#23> prio=10 os_prio=0 tid=0x00007fcc3804a800 nid=0x1a68 runnable [0x00007fcc42fcd000] java.lang.Thread.State: RUNNABLE at io.netty.channel.unix.FileDescriptor.writeAddress(Native Method) at io.netty.channel.unix.FileDescriptor.writeAddress(FileDescriptor.java:135) at io.netty.channel.epoll.AbstractEpollChannel.doWriteBytes(AbstractEpollChannel.java:290) at io.netty.channel.epoll.AbstractEpollStreamChannel.writeBytes(AbstractEpollStreamChannel.java:260) at io.netty.channel.epoll.AbstractEpollStreamChannel.doWriteSingle(AbstractEpollStreamChannel.java:472) at io.netty.channel.epoll.AbstractEpollStreamChannel.doWrite(AbstractEpollStreamChannel.java:457) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#327 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAa0QuMMKy_qZCPCDQhGGn25quEZvCYtks5sKmx9gaJpZM4AB8w8>.

senthilec566 · 2017-07-05T07:00:37Z

@normanmaurer - Attached SS for your reference ... Many times it crosses 80+ % .. but somehow in command line i see different result :( ..
ps -o '%cpu,%mem' -p 14442
%CPU %MEM
403 8.3

normanmaurer · 2017-07-05T07:03:48Z

@senthilec566 can you attach a jmh / your kit profile ?

senthilec566 · 2017-07-05T07:59:55Z

snapshot.zip

Attaching YourKit SnapShot.. Hope this helps!

senthilec566 · 2017-07-05T09:07:46Z

Dead Lock :-(..

Potential deadlock: frozen threads found

It seems that the following threads have not changed their stack for more than 10 seconds.
These threads are possibly (but not necessarily!) in a deadlock or hung.

epollEventLoopGroup-2-1 <--- Frozen for at least 58m 12s
io.netty.channel.epoll.Native.epollWait0(int, long, int, int) Native.java (native)
io.netty.channel.epoll.Native.epollWait(int, EpollEventArray, int) Native.java:111
io.netty.channel.epoll.EpollEventLoop.epollWait(boolean) EpollEventLoop.java:230
io.netty.channel.epoll.EpollEventLoop.run() EpollEventLoop.java:254
io.netty.util.concurrent.SingleThreadEventExecutor$5.run() SingleThreadEventExecutor.java:858
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run() DefaultThreadFactory.java:138
java.lang.Thread.run() Thread.java:748

senthilec566 · 2017-07-05T09:08:31Z

@normanmaurer - Any work around for this ? Should i try old version of Netty ??

senthilec566 · 2017-07-06T08:21:07Z

@normanmaurer - Any findings ? Meantime i will try other version of netty ..

Scottmitch · 2017-07-06T14:14:40Z

@senthilec566 - your issue is using the Epoll JNI transport but this issue was opened against the NIO transport. Can you open another issue?

Also what is your usage pattern look like? Do you have sockets being opened/closed constantly or just a few that are active? If you take successive snapshots (e..g 20 ish) do you see the stack trace change? Seeing the stack trace at epillWait is not necessarily a problem and can be common (it's how the OS tells us what events are pending and is called frequently). Also a reproducer would be nice.

umerpk · 2017-10-26T13:34:34Z

We are still having same issue on Java 1.8?

"GS-LRMI-Selector-Read-Thread-0" daemon prio=10 tid=0x00007fdc3d550800 nid=0x7533 runnable [0x00007fdbbafdd000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x0000000572484890> (a sun.nio.ch.Util$2)
- locked <0x00000005724848a0> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000572484848> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at com.gigaspaces.lrmi.nio.selector.handler.AbstractSelectorThread.doSelect(AbstractSelectorThread.java:62)
at com.gigaspaces.lrmi.nio.selector.handler.AbstractSelectorThread.run(AbstractSelectorThread.java:51)
at java.lang.Thread.run(Thread.java:744)

Locked ownable synchronizers:
- None

rohitsahay2000 · 2018-11-13T07:43:19Z

Facing the same issue with jdk 1.8, netty 4.1.25. The below screenshot was taken when the thread count already hit 1450 (starting from 200)

The event loop group thread count keeps on increasing leading to very high cpu usage and unresponsive container inspite of closing them regularly.

One of the typical threads (having lots of instances) from jstack
nioEventLoopGroup-8870-1 tid=13320
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:281)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:105)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:98)
- locked <0x4f8900cd> (a io.netty.channel.nio.SelectedSelectionKeySet)
- locked <0x83732742> (a java.util.Collections$UnmodifiableSet)
- locked <0x288f21af> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:109)
at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)
at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:756)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:411)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:811)

Locked ownable synchronizers:
- None

VaughnVernon · 2020-01-30T17:22:30Z

Hi All, We have hit this problem in a different project (not using Netty) and are looking for a fix/workaround. This only happens on Linux. We would really appreciate some help in learning how you solved this. Would someone please summarize the solution and source files involved? Sorry, it's difficult to piece together what finally worked given all the related discussions/issues.

vlingo/xoom-wire#28

atorr0 · 2020-04-08T14:29:33Z

Hi all,

The epoll bug is a well-known one with Java NIO framework, that seems fixed in NIO2.

I hope Netty not having to delve into JNI/JNA/etc. to fix it, but in this case maybe you can even improve it.

More info on #2515

Workeryuan · 2021-03-11T08:26:11Z

We are still having same issue on Development environment jdk1.8，Operating environment jre 1.2.
#27 daemon prio=5 os_prio=0 tid=0x00007f6cc41f3000 nid=0xbc59 runnable [0x00007f6cd69f4000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000006c08856c8> (a sun.nio.ch.Util$3)
- locked <0x00000006c08856d8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000006c0885680> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)

Locked ownable synchronizers:
- None

ITzhangqiang · 2021-07-21T03:20:32Z

so how to slove this issue？

Suddenlove · 2022-03-01T04:00:10Z

My application changed netty from 3.6.0.Final to 3.10.6.Final and CPU usage returns to normal, JDK version is 1.8.0_101.

happyforver · 2022-12-12T07:26:57Z

We are still having same issue on Development environment jdk1.8，Operating environment jre 1.2. #27 daemon prio=5 os_prio=0 tid=0x00007f6cc41f3000 nid=0xbc59 runnable [0x00007f6cd69f4000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x00000006c08856c8> (a sun.nio.ch.Util$3) - locked <0x00000006c08856d8> (a java.util.Collections$UnmodifiableSet) - locked <0x00000006c0885680> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)

Locked ownable synchronizers: - None

Workeryuan

hi, now, i have same issue. do you fix this ?

happyforver · 2022-12-13T18:21:36Z

i updated to jdk11 , not working...

ghost assigned normanmaurer Aug 23, 2012

normanmaurer added a commit that referenced this issue Aug 25, 2012

Re-create Selector if we hit the epoll(..) jdk bug which leads to 100…

ff3f2b6

…% cpu load. This is just a workaround but helps to recover. See #327

normanmaurer mentioned this issue Aug 25, 2012

Re-create Selector if we hit the epoll(..) jdk bug which leads to 100% c... #565

Merged

normanmaurer added a commit that referenced this issue Aug 29, 2012

Merge workaround for epoll(..) bug into master branch. See #365 , #535 …

3d44aec

…and #327

normanmaurer closed this as completed Aug 29, 2012

normanmaurer reopened this Aug 31, 2012

normanmaurer added a commit that referenced this issue Aug 31, 2012

Correctly detect if the Selector.select(..) unblocks because of a man…

63a234b

…ual wakup or because of a closed channel. See #327

normanmaurer added a commit that referenced this issue Aug 31, 2012

Address comments of @trustin and @kimchy. See #327

fee1492

normanmaurer added a commit that referenced this issue Aug 31, 2012

Handle CancelledKeyException. See #327

e2464a1

normanmaurer added a commit that referenced this issue Aug 31, 2012

Port latest fixes for the epoll(..) workaround. See #327

db98b9f

normanmaurer added a commit that referenced this issue Aug 31, 2012

Disable epoll bug workaround by default. It can be enabled via -Dorg.…

45829ff

…jboss.netty.epollBugWorkaround=true. It will be enabled by default later if it has proven to be stable. See #327

normanmaurer added a commit that referenced this issue Sep 3, 2012

Calculate the 80% the right way. See #327

a61403d

normanmaurer closed this as completed Sep 3, 2012

theorangesock mentioned this issue Sep 16, 2012

Kryonet doesn't recreate Selector with select spin detected svn2github/kryonet#1

Open

unbinder mentioned this issue Oct 17, 2012

After applying # 327 Workaround for EPollArrayWrapper.epollWait issue still 100 % CPU load #663

Closed

actiontech-bot mentioned this issue Aug 24, 2017

EPollArrayWrapper.epollWait 100% CPU actiontech/dble#11

Closed

ashijay mentioned this issue Oct 23, 2017

Epoll errors still appearing on Netty 3.9.9 version #7335

Closed

fredoboulo mentioned this issue Feb 9, 2018

IO thread loops infinitely on out event and blocks application with 100% CPU usage [was renamed] zeromq/jeromq#520

Closed

rohitsahay2000 mentioned this issue Nov 13, 2018

Added option to do busy-wait on epoll #8267

Merged

rohitsahay2000 mentioned this issue Nov 20, 2018

High CPU Usage with NioEventLoopGroup #8553

Closed

QilongZhang mentioned this issue Apr 13, 2019

SOFA-ARK-telnet-server-worker-0-T1”的线程CPU持续占用率100% sofastack/sofa-boot#382

Closed

ocswor mentioned this issue Sep 25, 2021

JAVE CPU 几乎占用100% master-coder-ll/v2ray-web-manager#169

Closed

expertjavaguy added a commit to expertjavaguy/AdamfiskProxy that referenced this issue May 26, 2022

upgrade netty to fix CPU usage in Lantern refs netty/netty#327

1702c33

EPollArrayWrapper.epollWait 100% CPU Usage #327

EPollArrayWrapper.epollWait 100% CPU Usage #327

Comments

blucas commented May 16, 2012

blucas commented May 16, 2012

blucas commented May 18, 2012

Eirenliel commented Jun 14, 2012

Eirenliel commented Jun 14, 2012

twinforces commented Jul 9, 2012

incubos commented Jul 26, 2012

vikiitd commented Aug 2, 2012

vikiitd commented Aug 8, 2012

trustin commented Aug 9, 2012

vrajesh1989 commented Aug 12, 2012

normanmaurer commented Aug 23, 2012

normanmaurer commented Aug 25, 2012

vikiitd commented Aug 27, 2012

normanmaurer commented Aug 27, 2012

normanmaurer commented Aug 29, 2012

zaffargachal commented Feb 22, 2017

senthilec566 commented Jul 4, 2017

senthilec566 commented Jul 4, 2017

doom369 commented Jul 4, 2017 • edited Loading

senthilec566 commented Jul 4, 2017

normanmaurer commented Jul 4, 2017 via email

senthilec566 commented Jul 5, 2017 • edited Loading

normanmaurer commented Jul 5, 2017

senthilec566 commented Jul 5, 2017

senthilec566 commented Jul 5, 2017

senthilec566 commented Jul 5, 2017

senthilec566 commented Jul 6, 2017

Scottmitch commented Jul 6, 2017

umerpk commented Oct 26, 2017

rohitsahay2000 commented Nov 13, 2018 • edited Loading

VaughnVernon commented Jan 30, 2020

atorr0 commented Apr 8, 2020

Workeryuan commented Mar 11, 2021

ITzhangqiang commented Jul 21, 2021

Suddenlove commented Mar 1, 2022

happyforver commented Dec 12, 2022 • edited Loading

happyforver commented Dec 13, 2022

doom369 commented Jul 4, 2017 •

edited

Loading

senthilec566 commented Jul 5, 2017 •

edited

Loading

rohitsahay2000 commented Nov 13, 2018 •

edited

Loading

happyforver commented Dec 12, 2022 •

edited

Loading