[Question] big gap before and after execution of implemented service logic

[jfeng]

Please answer these questions before submitting your issue.

What version of gRPC are you using?

1.23.0

What did you expect to see?

We are trying to implement a high-throughput low latency service, when increasing throughput, noticed increased amount of traffic has gaps before and/or after the execution of implemented logic, as shown in the screenshot below:

Here is grpc server side configs:

		ThreadFactory tfExecutor = new DefaultThreadFactory("cads-mle-elg-executor");
		int executorPerAvaialbleProcessor = 1;
		Executor executor = Executors.newFixedThreadPool(executorPerAvaialbleProcessor, tfExecutor); // set executorPerAvaialbelProcessor to 1, will ends up with number of executors equals to the number of cores

		ServerBuilder<?> nettyServerBuilder = NettyServerBuilder.forPort(Environment.PORT)
.flowControlWindow(NettyChannelBuilder.DEFAULT_FLOW_CONTROL_WINDOW * 2)
				.bossEventLoopGroup(bossEventLoopGroup)  // 1
				.workerEventLoopGroup(workerEventLoopGroup) // 2 * cores
				.channelType(NioServerSocketChannel.class)
				.executor(executor);

Wondering if anyone has insights about what those gaps are? or what the executor is waiting for?

[creamsoup]

gRPC ships with perfmark. it provides visibility of thread hopping in gRPC which is nice to debug those kind of problem.

[jfeng]

cool, thanks for the info! i'll try the tool out