Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cilium, bigtcp: Make probing for GRO/GSO max size more graceful #26385

Merged
merged 3 commits into from Jun 21, 2023
Merged

cilium, bigtcp: Make probing for GRO/GSO max size more graceful #26385

merged 3 commits into from Jun 21, 2023

Conversation

borkmann
Copy link
Member

See commits.

@borkmann
cilium, bigtcp: Misc comment improvements
a3f69b6 
Just some tiny tweaks, that's all.

Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
@borkmann borkmann added sig/datapath Impacts bpf/ or low-level forwarding details, including map management and monitor messages. release-note/misc This PR makes changes that have no direct user impact. labels Jun 20, 2023
@borkmann borkmann requested a review from a team as a code owner June 20, 2023 16:23
@borkmann borkmann requested a review from aspsk June 20, 2023 16:23
@borkmann
Copy link
Member Author

borkmann commented Jun 20, 2023 •
edited

Cc @haiyuewa, could you test this for ice case? Thanks!

@borkmann
cilium, bigtcp: Make probing for GRO/GSO max size more graceful
801e8d6 
This would in particular help drivers which do not have bigTCPGSOMaxSize
of 196608, for example, ice. Lower the maximum if BIG TCP is supported and
smaller than the default of 196608. We might make the bigTCP{GSO,GRO}MaxSize
also manually tweakable in future.

Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
@borkmann
Copy link
Member Author

/test

@haiyuewa
Copy link
Contributor

@borkmann Cool, small code, big improvement:

IPv4 BIG TCP, IPv6 BIG TCP disabled:
kubectl exec netperf-client -- netperf -t TCP_RR -H ${NETPERF_SERVER_IPV6} -- -r80000:80000 -O MIN_LATENCY,P90_LATENCY,P99_LATENCY,THROUGHPUT
MIGRATED TCP REQUEST/RESPONSE TEST from ::0 (::) port 0 AF_INET6 to 2013:2013:2013:1::acd2 () port 0 AF_INET6 : first burst 0
Minimum 90th 99th Throughput
Latency Percentile Percentile
Microseconds Latency Latency
Microseconds Microseconds
86 273 706 4782.05

kubectl exec netperf-client -- netperf -t TCP_RR -H ${NETPERF_SERVER_IPV4} -- -r80000:80000 -O MIN_LATENCY,P90_LATENCY,P99_LATENCY,THROUGHPUT
MIGRATED TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 13.13.1.97 (13.13.1) port 0 AF_INET : first burst 0
Minimum 90th 99th Throughput
Latency Percentile Percentile
Microseconds Latency Latency
Microseconds Microseconds
86 319 772 4441.95

============================================

IPv4 BIG TCP, IPv6 BIG TCP enabled:

kubectl -n kube-system logs ds/cilium 2>&1 | grep "big-tcp"
level=info msg=" --enable-ipv4-big-tcp='true'" subsys=daemon
level=info msg=" --enable-ipv6-big-tcp='true'" subsys=daemon
level=info msg="Setting up BIG TCP" subsys=big-tcp
level=info msg="Lowering GRO/GSO max size from 196608 to 131072" device=enp0 subsys=big-tcp
level=info msg="Setting IPv6 gso_max_size to 131072 and gro_max_size to 131072" device=enp0 subsys=big-tcp
level=info msg="Setting IPv4 gso_max_size to 131072 and gro_max_size to 131072" device=enp0 subsys=big-tcp

kubectl exec netperf-server -- ip -d -j link show dev eth0 | jq -c '.[0].gso_max_size'
131072

kubectl exec netperf-client -- ip -d -j link show dev eth0 | jq -c '.[0].gso_max_size'
131072

ip -d -j link show dev enp0 | jq -c '.[0].gso_max_size'
131072

ip -d -j link show dev enp0 | jq -c '.[0].gso_ipv4_max_size'
131072

kubectl exec netperf-client -- netperf -t TCP_RR -H ${NETPERF_SERVER_IPV6} -- -r80000:80000 -O MIN_LATENCY,P90_LATENCY,P99_LATENCY,THROUGHPUT
MIGRATED TCP REQUEST/RESPONSE TEST from ::0 (::) port 0 AF_INET6 to 2013:2013:2013:1::a041 () port 0 AF_INET6 : first burst 0
Minimum 90th 99th Throughput
Latency Percentile Percentile
Microseconds Latency Latency
Microseconds Microseconds
75 158 651 7383.83

kubectl exec netperf-client -- netperf -t TCP_RR -H ${NETPERF_SERVER_IPV4} -- -r80000:80000 -O MIN_LATENCY,P90_LATENCY,P99_LATENCY,THROUGHPUT
MIGRATED TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 13.13.1.177 (13.13.1) port 0 AF_INET : first burst 0
Minimum 90th 99th Throughput
Latency Percentile Percentile
Microseconds Latency Latency
Microseconds Microseconds
74 146 453 7791.12

@borkmann
Copy link
Member Author

Awesome, thanks @haiyuewa!

@borkmann
cilium, bigtcp: Add ice to the list of supported drivers
0e70f06 
Now that we support lowering GRO/GSO, we can also add ice.

Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Tested-by: Haiyue Wang <haiyue.wang@intel.com>
@borkmann
Copy link
Member Author

(CI all green, no code change. Adding a small doc update to list ice.)

@borkmann borkmann requested a review from a team as a code owner June 21, 2023 06:12
@borkmann borkmann requested a review from learnitall June 21, 2023 06:12
@borkmann borkmann added the ready-to-merge This PR has passed all tests and received consensus from code owners to merge. label Jun 21, 2023
@borkmann borkmann merged commit ae13023 into main Jun 21, 2023
51 of 52 checks passed
@borkmann borkmann deleted the pr/bigtcp-probe branch June 21, 2023 06:32
@joestringer joestringer mentioned this pull request Jun 28, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@aspsk aspsk aspsk approved these changes

@learnitall learnitall Awaiting requested review from learnitall learnitall is a code owner automatically assigned from cilium/docs-structure

Assignees
No one assigned
Labels
ready-to-merge This PR has passed all tests and received consensus from code owners to merge. release-note/misc This PR makes changes that have no direct user impact. sig/datapath Impacts bpf/ or low-level forwarding details, including map management and monitor messages.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@borkmann @haiyuewa @aspsk