High latency establishing new connection and sending first request

[nkrode]

I'm seeing extremely high latency establishing new connection and sending in the first request (simple ping/pong). It's taking ~ 80 ms to just do the initial handshake,HTTP2 settings exchange etc.

Here's a Wireshark capture of the packets between a grpc client and server running on the same host and you can see the actual RPC request is made ~ 100ms.

Not sure If I have something misconfigured, just wanted to confirm if this is normal?

What version of gRPC are you using?

1.12.0

[carl-mastrangelo]

Is it possible that you are measuring the Java running in interpretted mode? Usually the JIT doesn't kick in for this stuff until it has been under load for some time.

[nkrode]

No, I tested after warming up the server and client with close to 500k iterations over 10 mins period.

Ignoring the latency, currently I'm curious if wireshark capture shows redundant packets being exchanged? Is it normal to exchange WINDOW_UPDATE and SETTINGS 2-3 times between the server and client before the RPC request is made?

[carl-mastrangelo]

Those frames are normal, since each side maintains its own receive window, and settings. Java gRPC uses a different settings than Netty (we set the default window size to 1MB iirc). Best thing I can suggest is to open tons of connection and attach a profiler (like jprofiler or yourkit). It's really the only way to know what it's doing.

[nkrode]

👍 closing

[nkrode]

I tested this with HelloWorld Server and client from examples. Looks the same, takes about 68ms to issue the RPC call, admittedly I didn't warm up the code, no JIT kicking in, even then a 68ms handshake time to establish a connection on cold start seems excessively high, don't you think?

[carl-mastrangelo]

I don't think so. The class loading and initialization requires disk access time, syscalls, etc. You might consider running strace on your binary to see when exactly the reads and writes are being fired. You can also use the perf tool to measure things like page faults and cycle counters. Honestly I don't think wireshark is the ideal tool to diagnose this.

[nkrode]

Sounds good, will try those.