Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

vmware: Disable UDP offload for primary interface #2850

Merged
merged 1 commit into from
Mar 6, 2023

Conversation

zmrow
Copy link
Contributor

@zmrow zmrow commented Mar 3, 2023

Issue number:
N/A
Depends on #2829

Description of changes:
We've had reports of Bottlerocket in VMware having networking issues that we've traced back to the combination of our 5.10 kernel and the vmxnet3 driver. This driver enables UDP offload which has caused conflicts with NSX-T in certain environments.

This PR adds a systemd unit to VMware variants only. The systemd unit disables UDP offload via ethtool commands for the default primary interface in VMware: eth0.

The unit does reference eth0. Given that eth0 is the default primary interface in VMware, and changing it isn't trivial (especially with EKS-A), this felt pretty safe and should cover the 98% use case.

If a user does opt to do their own networking shenanigans, and runs into an issue where UDP offload is a problem, they are able to use a bootstrap container to run this same set of commands for their preferred interface.

Testing done:

  • Build an aws-k8s-1.24 image and ensure the unit does not exist and is not run
bash-5.1# systemctl --type=service | grep -i disable-udp
bash-5.1#
  • Build a vmware-k8s-1.24 image and ensure the unit runs and the settings are applied for the interface.
bash-5.1# systemctl --type=service | grep -i disable-udp
  disable-udp-offload.service            loaded active     exited          Disables UDP offload

bash-5.1# ethtool -k eth0        
Features for eth0:
...
tx-udp_tnl-segmentation: off
tx-udp_tnl-csum-segmentation: off
...

Without the systemd unit - the settings show as "on":

bash-5.1# ethtool -k eth0 | grep -i tx-udp
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: on

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

@zmrow zmrow requested review from bcressey and yeazelm March 3, 2023 01:06
@bcressey
Copy link
Contributor

bcressey commented Mar 3, 2023

In VMware, Bottlerocket's 5.10 kernel and the vmxnet3 driver can conflict with NSX-T in certain environments. Disabling UDP offload works around the problem.

Couple notes:

  • 5.15 kernel is also affected
  • NSX / NSX-T reportedly is not involved

Please revise the PR description and commit message accordingly.

packages/os/disable-udp-offload.service Outdated Show resolved Hide resolved
packages/os/disable-udp-offload.service Show resolved Hide resolved
packages/os/disable-udp-offload.service Outdated Show resolved Hide resolved
packages/os/os.spec Outdated Show resolved Hide resolved
packages/os/disable-udp-offload.service Show resolved Hide resolved
In VMware, the Geneve / VXLan tunnels used by Cilium affect the vmxnet3
driver in recent kernels and can cause issues.  Disabling UDP offload
works around this problem.

This commit adds a systemd unit to VMware variants only.  The systemd
unit disables UDP offload for the default primary interface (eth0) via
ethtool commands.

The unit is required by preconfigured.target and runs before bootstrap
containers, which allows the user the chance to revert the change in a
bootstrap container if they see fit.
@zmrow
Copy link
Contributor Author

zmrow commented Mar 6, 2023

^ Addresses @bcressey 's comments.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants