New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[bdp] Maximum value limits throughput on high-latency connections #2400

Closed
euroelessar opened this Issue Oct 23, 2018 · 4 comments

Comments

Projects
None yet
2 participants
@euroelessar
Copy link

euroelessar commented Oct 23, 2018

What version of gRPC are you using?

grpc-go 1.14 (but present on master as well)

What did you do?

Use server-side streaming (single response stream over single connection) and measure throughput.
The latency between two hosts is 100ms.

What did you expect to see?

Performance is limited by network stack or CPU, and is on par with iperf.

What did you see instead?

Performance is limited by 40MB/s. Both CPU and network are under-utilized.

gRPC network stack quickly reaches limit of 4MB window size and doesn't scale it further.

Manually specifying both InitialWindowSize and InitialConnWindowSize, or modifying grpc-go codebase to increase bdpLimit resolves the performance issue as well, but it doesn't scale well for busy or otherwise low-throughput network.

@dfawley

This comment has been minimized.

Copy link
Contributor

dfawley commented Oct 23, 2018

@euroelessar the 4MB window size limit is set based on standard TCP settings -- most TCP connections can't effectively utilize any larger of a window size. What is the discrepancy between our out-of-the-box 40MB/s and what you're seeing with iperf or when you set Initial[Conn]WindowSize? What iperf tests are you running when you see these results?

@euroelessar

This comment has been minimized.

Copy link

euroelessar commented Oct 23, 2018

Currently the difference is 40MBps vs 100MBps for cross-zone traffic (≈150% delta), and 300MBps vs 450MBps for traffic within (≈50% delta). But within a metro we're likely hitting CPU utilization bottleneck at 450MBps, as iperf can do ~1.2GBps over single TCP connection.
It does worth noting that this numbers are for gRPC over TLS connection.

Regarding iperf test:
server: $ iperf iperf -s -p 6000
client: $ iperf -c ${remote_server} -p 6000 -P 1
output:

[  5] local $my port 6000 connected with $remote port 26460
[  5]  0.0-10.1 sec  1004 MBytes   838 Mbits/sec
[  4] local $my port 6000 connected with $local port 51102
[  4]  0.0-10.0 sec  10.7 GBytes  9.22 Gbits/sec
@dfawley

This comment has been minimized.

Copy link
Contributor

dfawley commented Oct 24, 2018

Looks like 4MB is definitely too conservative of a default value - we are open to increasing this, or possibly reading the TCP settings and mirroring the receive window size when setting our flow control.

@euroelessar could you do a cat /proc/sys/net/ipv4/tcp_rmem so we can see what your TCP receive window size can scale to (on both the client and server if they are not homogeneous)? Thanks!

@euroelessar

This comment has been minimized.

Copy link

euroelessar commented Oct 24, 2018

Sure,

$ cat /proc/sys/net/ipv4/tcp_rmem
4096	135168	16777216
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment