what is the best practice to handle such kind of traffic?

[Mr-Giraffe]

Please answer these questions before submitting your issue.

What version of gRPC are you using?

1.0.0

What JVM are you using (java -version)?

JDK1.7.0_79

What did you do?

when I run top, I found about 100 thread are running(or waiting) in our GRPC server.

Threads: 98 total, 6 running, 92 sleeping, 0 stopped, 0 zombie
%Cpu(s): 28.4 us, 5.0 sy, 0.0 ni, 66.1 id, 0.0 wa, 0.0 hi, 0.5 si, 0.0 st
KiB Mem: 12305560 total, 6309616 used, 5995944 free, 141816 buffers
KiB Swap: 0 total, 0 used, 0 free. 2453180 cached Mem

What did you expect to see?

I am not sure this is normal. Will this cause too much context switch?

What did you see instead?

As above.

More details about our application

We build a queue service with gRPC. A bout 1000 nodes(VM) are sending request to our gRPC server with "while(true)" and only several of them could get the real traffic.

We choose the default server, i.e. netty to serve that kind of traffic. So, what is the proper config or coding we should use?

Another question is that we shared the "ManagedChannel" object in about 200 nodes(they may use this object concurrently) and build the stub in each call. Is this the right way to use?

[ejona86]

There are two main thread pools in grpc: Netty event loops and application executor. The Netty event loops are current 2x the number of cores. The application executor is a Executors.newCachedThreadPool(), so it scales with load (possibly to its detriment). It should not end up being more than the maximum number of concurrent RPCs. It can be a good idea to override the application executor, especially on server-side, via ServerBuilder.executor() (ManagedChannelBuilder.executor() for client-side).

How many cores are on your machine?

Another question is that we shared the "ManagedChannel" object in about 200 nodes(they may use this object concurrently) and build the stub in each call. Is this the right way to use?

Yep. That sounds great. Both Channels and Stubs can be shared, and we would encourage you to when you're easily able.

[Mr-Giraffe]

@ejona86 Thanks for the feedback.

From my understanding, cachedTreadPool is very suitable for short live model, and our calls are do short live.. Maybe we need to expose more metrics on this part.

The CPU related info is as follows:

$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 1
Core(s) per socket: 1
Socket(s): 4
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 60
Stepping: 1
CPU MHz: 2097.570
BogoMIPS: 4195.14
Virtualization: VT-x
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 32K
L1i cache: 32K
L2 cache: 4096K
NUMA node0 CPU(s): 0-3

[ejona86]

Are you sure all those threads are from Java? It looks like you just pasted the output from top, which includes kernel threads and daemons. On a pretty minimal server I maintain (with 500 MB of RAM, no Java) I see 97 threads and on a small ARM server I see 130.

What does pstree show or a Java thread dump show?

[Mr-Giraffe]

Sorry, I missed some detail. Actually, I run "top" first to find the PID and then run "top - H -p $Pid" to see the whole threading info.

Here is the output of pstree:

init─┬─
        ├─java───98*[{java}]

[ejona86]

@Mr-Giraffe, oh, okay. I think I'd just suggest taking a thread dump at this point to see where all the threads are coming from.

[Mr-Giraffe]

@ejona86 By doing more investigations, I found that we have a leak of the following objects:
io.netty.handler.codec.http2.DefaultHttp2Connection$DefaultStream

I tried to read part of the source code and from my current understanding, the Stream object could kind of mapping to "Calls". For each call, it will generate 1 stream and 1 stream belongs to 1 connection. The count of Stream object could be mapped to the on going calls. Am I right?

Our GRPC server is behind a gRPC-Gateway. The connection between gPRC-Gateway and our gRPc server is about 500. But the count of streams in our memory becomes bigger and bigger. We add some metrics to the gRPC server and found that the on going calls are only 300.(doing this by add an server interceptor, count when calls in and when calls close method).

Part of our service nodes suffer a full GC, and the count of DefaultStream is about 1914000 in a 4G JVM.

So, could you please share more about how the requests are handled in gRPC?

[ejona86]

I found that we have a leak of the following objects:
io.netty.handler.codec.http2.DefaultHttp2Connection$DefaultStream

You're on grpc-java v1.0.0 now? Try upgrading to v1.0.2, which bumped Netty to v4.1.6 and fixed #2358. I would generally encourage tracking our new releases; there's been a lot of fixes in both grpc-java and Netty.

I tried to read part of the source code and from my current understanding, the Stream object could kind of mapping to "Calls". For each call, it will generate 1 stream and 1 stream belongs to 1 connection. The count of Stream object could be mapped to the on going calls. Am I right?

Yeah, that's right. In the future, once we have retries a Call could have multiple streams, but that's not too surprising.

Part of our service nodes suffer a full GC, and the count of DefaultStream is about 1914000 in a 4G JVM.

If v1.0.2 doesn't fix you, then it'd be good to know what is retaining those streams.

[Mr-Giraffe]

@ejona86 thanks for the reply.

I tired 1.0.3 version last week and it seems that the issue is still there. Some JVMs in our pool are continu to full GC and could not serve the traffic properly.

One of my concern is that we throw StatusRuntimeException in the Interceptor(ununthenticated header) and not sure this is the right way or may cause the memory leak issue.

The JVM which is not frequently full GC also has 420, 000+ DefaultStreams object. If the count of these object should be eaqual to the on going request, it is too much also.

Could you please offer us some hint to find the object that may hold the Stream object? The logic in our pool is really simple: hold some stubs to call another gRPC service. The stubs and channels in our pool are all shared as you suggested.

[ejona86]

I tired 1.0.3 version last week and it seems that the issue is still there.

Then you may need to upgrade to 1.1 or later. 1.0.x is really old at this point, and there's been lots of fixes.

One of my concern is that we throw StatusRuntimeException in the Interceptor(ununthenticated header) and not sure this is the right way or may cause the memory leak issue.

This should not cause a leak. But you should be aware of #2189.

The JVM which is not frequently full GC also has 420, 000+ DefaultStreams object. If the count of these object should be eaqual to the on going request, it is too much also.

Those are very low-level Netty objects. Each NettyClientStream should correspond to one DefaultStream, so that's normal.

Could you please offer us some hint to find the object that may hold the Stream object?

I'd suggest taking a heap dump and then looking at it in a tool like jvisualvm. Computing retained sizes generally sheds a lot of light quickly. You can also start at some low-level objects and trace how they are retained. But honestly, I wouldn't bother until I upgraded to the latest release.