Netty ObjectCleanerThread keeps high CPU

[thermosym]

Please answer these questions before submitting your issue.

What version of gRPC are you using?

1.11.0

What did you expect to see?

One Netty thread keeps high CPU usage. The following is highest CPU usage related jstack info

"ObjectCleanerThread" #28 daemon prio=1 os_prio=0 tid=0x00007f19c6fb1800 nid=0x1a4e runnable [0x00007f193219d000]
java.lang.Thread.State: RUNNABLE
at io.netty.util.Recycler$WeakOrderQueue$Head.run(Recycler.java:264)
at io.netty.util.internal.ObjectCleaner$AutomaticCleanerReference.cleanup(ObjectCleaner.java:143)
at io.netty.util.internal.ObjectCleaner$1.run(ObjectCleaner.java:62)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)

[carl-mastrangelo]

@thermosym Approximately how many RPCs per second are you handling? Also, are you using OpenJDK?

[thermosym]

@carl-mastrangelo Actually the load is not high, approximately 10 rpc/s.
I'm using Oracle JDK 1.8.0_161
with JVM parameter: -Xmx1024M -XX:+UseG1GC -XX:MaxGCPauseMillis=200
The memory monitoring result shows that JVM GC works fine, seems no memory leak.

[carl-mastrangelo]

That's unusual, Are you running with any custom recycler flags? Like io.netty.recycler.maxCapacityPerThread or io.netty.recycler.maxSharedCapacityFactor ?

Also, there was a change in netty to make the recycler use a dedicated cleaner thread, but I don't recall which version of Netty it was in. (1.11.0 uses Netty 4.1.22).
@normanmaurer would have a better idea of what debugging info would be needed to solve this.

[normanmaurer]

@carl-mastrangelo this may be related to netty/netty@3ec2945 which is included since 4.1.24.Final ?

[carl-mastrangelo]

@normanmaurer That may be it. We are still on 4.1.22 on our current release. I can bump us to 4.1.25, In the mean time, @thermosym can you use gRPC 1.10.0 ?

[normanmaurer]

@carl-mastrangelo this looks more like netty 4.1.23.Final which has the bug I mentioned before:

https://github.com/netty/netty/blob/netty-4.1.23.Final/common/src/main/java/io/netty/util/Recycler.java#L264

[normanmaurer]

So just to make it clear this is fixed in 4.1.24.Final but I would just use 4.1.25.Final directly

[carl-mastrangelo]

@normanmaurer thanks for the input. I bumped to 4.1.25

[thermosym]

Thanks @carl-mastrangelo @normanmaurer for the help, I'll try to use version 1.10.1