Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

VLAN interfaces do not get offload enabled #6629

Closed
1 task done
bobrik opened this issue Aug 17, 2017 · 10 comments
Closed
1 task done

VLAN interfaces do not get offload enabled #6629

bobrik opened this issue Aug 17, 2017 · 10 comments
Labels
bug 🐛 Programming errors, that need preferential fixing needs-reporter-feedback ❓ There's an unanswered question, the reporter needs to answer network udev

Comments

@bobrik
Copy link

bobrik commented Aug 17, 2017

Submission type

  • Bug report

systemd version the issue has been seen with

systemd 232
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN

Used distribution

Debian Stretch

In case of bug report: Expected behaviour you didn't see

Offload features are enabled for VLAN interfaces.

In case of bug report: Unexpected behaviour you saw

Offload features are not enabled for VLAN interfaces.

In case of bug report: Steps to reproduce the problem

Recently we upgraded some machines from Debian Jessie to Debain Stretch and noticed that system cpu jumped by quite a lot, even though we use exactly the same kernel and have exactly the same workloads. In some cases machines were so overwhelmed, they got stalled page allocations for over a minute and all other stuff you get when it comes to this.

These machines are stateless, so we were able to rollback and see performance improvements.

Tracing revealed that system CPU time was wasted in timers started from TCP subsystem. In the end we were able to pin point the difference to enabled features of vlan interface that was getting all the traffic.

We have vlan interface on top of bonded interface, all interfaces are configured with systemd-networkd exactly the same between distributions. Bond is correctly configured, while vlan is not.

Jessie with systemd v230 from backports:

# ethtool -k bond0
Features for bond0:
rx-checksumming: off [fixed]
tx-checksumming: on
	tx-checksum-ipv4: off [fixed]
	tx-checksum-ip-generic: on
	tx-checksum-ipv6: off [fixed]
	tx-checksum-fcoe-crc: off [fixed]
	tx-checksum-sctp: off [fixed]
scatter-gather: on
	tx-scatter-gather: on
	tx-scatter-gather-fraglist: off [requested on]
tcp-segmentation-offload: on
	tx-tcp-segmentation: on
	tx-tcp-ecn-segmentation: on
	tx-tcp-mangleid-segmentation: on
	tx-tcp6-segmentation: on
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: on
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: on
rx-vlan-filter: on
vlan-challenged: off [fixed]
tx-lockless: on [fixed]
netns-local: on [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: on
tx-gre-csum-segmentation: on
tx-ipxip4-segmentation: on
tx-ipxip6-segmentation: on
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: on
tx-gso-partial: off [fixed]
tx-sctp-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
busy-poll: off [fixed]
hw-tc-offload: off [fixed]

# ethtool -k vlan10
Features for vlan10:
rx-checksumming: off [fixed]
tx-checksumming: on
	tx-checksum-ipv4: off [fixed]
	tx-checksum-ip-generic: on
	tx-checksum-ipv6: off [fixed]
	tx-checksum-fcoe-crc: off
	tx-checksum-sctp: off
scatter-gather: on
	tx-scatter-gather: on
	tx-scatter-gather-fraglist: off
tcp-segmentation-offload: on
	tx-tcp-segmentation: on
	tx-tcp-ecn-segmentation: on
	tx-tcp-mangleid-segmentation: on
	tx-tcp6-segmentation: on
udp-fragmentation-offload: on
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: off [fixed]
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: on
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: on [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off
tx-gre-segmentation: off [fixed]
tx-gre-csum-segmentation: off [fixed]
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-udp_tnl-csum-segmentation: off [fixed]
tx-gso-partial: off [fixed]
tx-sctp-segmentation: on
fcoe-mtu: off
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
busy-poll: off [fixed]
hw-tc-offload: off [fixed]
  • Stretch with systemd v232:
# ethtool -k bond0
Features for bond0:
rx-checksumming: off [fixed]
tx-checksumming: on
	tx-checksum-ipv4: off [fixed]
	tx-checksum-ip-generic: on
	tx-checksum-ipv6: off [fixed]
	tx-checksum-fcoe-crc: off [fixed]
	tx-checksum-sctp: off [fixed]
scatter-gather: on
	tx-scatter-gather: on
	tx-scatter-gather-fraglist: off [requested on]
tcp-segmentation-offload: on
	tx-tcp-segmentation: on
	tx-tcp-ecn-segmentation: on
	tx-tcp-mangleid-segmentation: on
	tx-tcp6-segmentation: on
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [requested on]
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: on
rx-vlan-filter: on
vlan-challenged: off [fixed]
tx-lockless: on [fixed]
netns-local: on [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: on
tx-gre-csum-segmentation: on
tx-ipxip4-segmentation: on
tx-ipxip6-segmentation: on
tx-udp_tnl-segmentation: on
tx-udp_tnl-csum-segmentation: on
tx-gso-partial: off [fixed]
tx-sctp-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
busy-poll: off [fixed]
hw-tc-offload: off [fixed]

# ethtool -k vlan10
Features for vlan10:
rx-checksumming: off [fixed]
tx-checksumming: off
	tx-checksum-ipv4: off [fixed]
	tx-checksum-ip-generic: off
	tx-checksum-ipv6: off [fixed]
	tx-checksum-fcoe-crc: off
	tx-checksum-sctp: off
scatter-gather: off
	tx-scatter-gather: off
	tx-scatter-gather-fraglist: off
tcp-segmentation-offload: off
	tx-tcp-segmentation: off [requested on]
	tx-tcp-ecn-segmentation: off [requested on]
	tx-tcp-mangleid-segmentation: off [requested on]
	tx-tcp6-segmentation: off [requested on]
udp-fragmentation-offload: off [requested on]
generic-segmentation-offload: off [requested on]
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: off [fixed]
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: on
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: on [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off
tx-gre-segmentation: off [fixed]
tx-gre-csum-segmentation: off [fixed]
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-udp_tnl-csum-segmentation: off [fixed]
tx-gso-partial: off [fixed]
tx-sctp-segmentation: on
fcoe-mtu: off
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
busy-poll: off [fixed]
hw-tc-offload: off [fixed]

This is a little bit hard to grasp, so here's the diff:

  • bond0:
--- bond0.jessie.txt	2017-08-16 22:12:35.000000000 -0700
+++ bond0.stretch.txt	2017-08-16 22:11:23.000000000 -0700
@@ -18,7 +18,7 @@
 udp-fragmentation-offload: off [fixed]
 generic-segmentation-offload: on
 generic-receive-offload: on
-large-receive-offload: on
+large-receive-offload: off [requested on]
 rx-vlan-offload: on
 tx-vlan-offload: on
 ntuple-filters: off [fixed]
  • vlan10:
--- vlan10.jessie.txt	2017-08-16 22:11:40.000000000 -0700
+++ vlan10.stretch.txt	2017-08-16 22:14:06.000000000 -0700
@@ -1,22 +1,22 @@
 # ethtool -k vlan10
 Features for vlan10:
 rx-checksumming: off [fixed]
-tx-checksumming: on
+tx-checksumming: off
 	tx-checksum-ipv4: off [fixed]
-	tx-checksum-ip-generic: on
+	tx-checksum-ip-generic: off
 	tx-checksum-ipv6: off [fixed]
 	tx-checksum-fcoe-crc: off
 	tx-checksum-sctp: off
-scatter-gather: on
-	tx-scatter-gather: on
+scatter-gather: off
+	tx-scatter-gather: off
 	tx-scatter-gather-fraglist: off
-tcp-segmentation-offload: on
-	tx-tcp-segmentation: on
-	tx-tcp-ecn-segmentation: on
-	tx-tcp-mangleid-segmentation: on
-	tx-tcp6-segmentation: on
-udp-fragmentation-offload: on
-generic-segmentation-offload: on
+tcp-segmentation-offload: off
+	tx-tcp-segmentation: off [requested on]
+	tx-tcp-ecn-segmentation: off [requested on]
+	tx-tcp-mangleid-segmentation: off [requested on]
+	tx-tcp6-segmentation: off [requested on]
+udp-fragmentation-offload: off [requested on]
+generic-segmentation-offload: off [requested on]
 generic-receive-offload: on
 large-receive-offload: off [fixed]
 rx-vlan-offload: off [fixed]

cc @ssahani
@ssahani
Copy link
Contributor

ssahani commented Aug 17, 2017

Please see if this fixes.

commit 45d34fa7fb9f587f09e8cca882966050b4cbb5be
Author: Maks Naumov <maksqwe1@ukr.net>
Date:   Mon Nov 14 12:33:43 2016 +0200

    link: fix offload features initialization (#4639)

index 4578d0d9b..1dca37527 100644
--- a/src/udev/net/link-config.c
+++ b/src/udev/net/link-config.c
@@ -169,7 +169,7 @@ static int load_link(link_config_ctx *ctx, const char *filename) {
         link->duplex = _DUP_INVALID;
         link->autonegotiation = -1;
 
-        memset(&link->features, -1, _NET_DEV_FEAT_MAX);
+        memset(&link->features, -1, sizeof(link->features));
 
         r = config_parse(NULL, filename, file,
                          "Match\0Link\0Ethernet\0",

@bobrik
Copy link
Author

bobrik commented Aug 17, 2017

@ssahani Debian patches: http://sources.debian.net/patches/systemd/232-25%2Bdeb9u1/

We have 232-25+deb9u1 installed, so we should have this patch:

It seems that the issue was fixed for everything except for VLAN links.

@poettering
Copy link
Member

@ssahani this appears to be something for you? Any idea?

@poettering poettering added network regression ⚠️ A bug in something that used to work correctly and broke through some recent commit labels Aug 29, 2017
@ssahani
Copy link
Contributor

ssahani commented Aug 30, 2017

I can't reproduce with my set up. I created vlan using ip link v230 networkd and the latest all result are same.

ip link

Features for trust:
rx-checksumming: off [fixed]
tx-checksumming: on
	tx-checksum-ipv4: off [fixed]
	tx-checksum-ip-generic: on
	tx-checksum-ipv6: off [fixed]
	tx-checksum-fcoe-crc: off [requested on]
	tx-checksum-sctp: off [requested on]
scatter-gather: on
	tx-scatter-gather: on
	tx-scatter-gather-fraglist: off [requested on]
tcp-segmentation-offload: on
	tx-tcp-segmentation: on
	tx-tcp-ecn-segmentation: on
	tx-tcp-mangleid-segmentation: on
	tx-tcp6-segmentation: on
udp-fragmentation-offload: on
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: off [fixed]
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: on
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: on [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [requested on]
tx-gre-segmentation: off [fixed]
tx-gre-csum-segmentation: off [fixed]
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-udp_tnl-csum-segmentation: off [fixed]
tx-gso-partial: off [fixed]
tx-sctp-segmentation: on
tx-esp-segmentation: off [fixed]
fcoe-mtu: off [requested on]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off [fixed]
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]

networkd (latest git)

Features for trust:
rx-checksumming: off [fixed]
tx-checksumming: on
	tx-checksum-ipv4: off [fixed]
	tx-checksum-ip-generic: on
	tx-checksum-ipv6: off [fixed]
	tx-checksum-fcoe-crc: off [requested on]
	tx-checksum-sctp: off [requested on]
scatter-gather: on
	tx-scatter-gather: on
	tx-scatter-gather-fraglist: off [requested on]
tcp-segmentation-offload: on
	tx-tcp-segmentation: on
	tx-tcp-ecn-segmentation: on
	tx-tcp-mangleid-segmentation: on
	tx-tcp6-segmentation: on
udp-fragmentation-offload: on
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: off [fixed]
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: on
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: on [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [requested on]
tx-gre-segmentation: off [fixed]
tx-gre-csum-segmentation: off [fixed]
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-udp_tnl-csum-segmentation: off [fixed]
tx-gso-partial: off [fixed]
tx-sctp-segmentation: on
tx-esp-segmentation: off [fixed]
fcoe-mtu: off [requested on]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off [fixed]
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]

networkd 230

Features for trust:
rx-checksumming: off [fixed]
tx-checksumming: on
	tx-checksum-ipv4: off [fixed]
	tx-checksum-ip-generic: on
	tx-checksum-ipv6: off [fixed]
	tx-checksum-fcoe-crc: off [requested on]
	tx-checksum-sctp: off [requested on]
scatter-gather: on
	tx-scatter-gather: on
	tx-scatter-gather-fraglist: off [requested on]
tcp-segmentation-offload: on
	tx-tcp-segmentation: on
	tx-tcp-ecn-segmentation: on
	tx-tcp-mangleid-segmentation: on
	tx-tcp6-segmentation: on
udp-fragmentation-offload: on
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: off [fixed]
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: on
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: on [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [requested on]
tx-gre-segmentation: off [fixed]
tx-gre-csum-segmentation: off [fixed]
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-udp_tnl-csum-segmentation: off [fixed]
tx-gso-partial: off [fixed]
tx-sctp-segmentation: on
tx-esp-segmentation: off [fixed]
fcoe-mtu: off [requested on]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off [fixed]
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]

@bobrik
Copy link
Author

bobrik commented Oct 16, 2017

@ssahani can you try systemd v232 that comes with Debian Stretch?

@ssahani
Copy link
Contributor

ssahani commented Nov 10, 2017

Sorry Only current version please.

@paulmenzel
Copy link
Contributor

Is there a corresponding Debian bug report?

@nevgeniev
Copy link

nevgeniev commented May 8, 2020 •
edited
Loading

guys i do have similar issue with rhel7 & onload which uses systemd 219

 $ ethtool -k p3p1 |grep vlan
rx-vlan-offload: off [fixed]
tx-vlan-offload: on [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]

can someone at least tell me which version of systemd should be vlan safe ?

@yuwata yuwata added bug 🐛 Programming errors, that need preferential fixing udev labels Nov 13, 2022
@yuwata yuwata added bug 🐛 Programming errors, that need preferential fixing and removed bug 🐛 Programming errors, that need preferential fixing regression ⚠️ A bug in something that used to work correctly and broke through some recent commit labels May 24, 2023
@yuwata
Copy link
Member

yuwata commented May 24, 2023

I cannot follow this issue. Is this really caused by systemd, rather than kernel? Why do you think so?

  • Please provide corresponding .network, .netdev, and .link files.
  • Please clarify the expected result and actual result.
  • Please provide debugging logs of networkd and udevd.

@yuwata yuwata added the needs-reporter-feedback ❓ There's an unanswered question, the reporter needs to answer label May 24, 2023
@bobrik
Copy link
Author

bobrik commented Jan 22, 2024

I replaced the workaround in our fleet with a check and a failure if this is detected. We rebooted thousands of machines and there were no issues.

Whatever the issue was back in the day, we don't see it today, so let's just close it.

@bobrik bobrik closed this as completed Jan 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug 🐛 Programming errors, that need preferential fixing needs-reporter-feedback ❓ There's an unanswered question, the reporter needs to answer network udev
Milestone
No milestone
Development

No branches or pull requests
6 participants
@bobrik @ssahani @poettering @nevgeniev @paulmenzel @yuwata