network: Introduce SR-IOV #16219

ssahani · 2020-06-19T07:17:00Z

cc @yuwata . I spent some time on SR-IOV. Here is the working version .

[Match]
Name=eni5np1

[SRIOV]
VirtualFunction=1
VLanId=5
QualityOfService=1
MACSpoofCheck=1
QueryReceiveSideScaling=1
Trust=1
LinkState=on

lgtm-com · 2020-06-19T13:37:52Z

This pull request introduces 1 alert when merging f4cbeb6 into a51a324 - view on LGTM.com

new alerts:

1 for Use of potentially dangerous function

yuwata · 2020-06-22T05:21:35Z

I will slightly update the commits and add tests for this.

yuwata · 2020-06-22T10:51:36Z

@ssahani Updated. PTAL.

ssahani · 2020-06-22T11:32:01Z

test/test-network/systemd-networkd-tests.py

+        return func
+
+    return f
+
 def expectedFailureIfCAKEIsNotAvailable():
    def f(func):


➜ ~ echo "99 2" > /sys/bus/netdevsim/new_device ➜ ~ ip link 9: eni99np1: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether f6:e7:ab:f4:5b:0d brd ff:ff:ff:ff:ff:ff 10: eni99np2: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether 16:0a:ff:96:bf:77 brd ff:ff:ff:ff:ff:ff ➜ ~ udevadm info -w10s /sys/devices/netdevsim99/net/eni99np1 P: /devices/netdevsim99/net/eni99np1 L: 0 E: DEVPATH=/devices/netdevsim99/net/eni99np1 E: SUBSYSTEM=net E: INTERFACE=eni99np1 E: IFINDEX=9 E: USEC_INITIALIZED=53327702503 E: ID_NET_NAMING_SCHEME=v245 E: ID_NET_NAME_PATH=eni99np1 E: ID_NET_DRIVER=netdevsim E: ID_NET_LINK_FILE=/usr/lib/systemd/network/99-default.link E: ID_NET_NAME=eni99np1 E: SYSTEMD_ALIAS=/sys/subsystem/net/devices/eni99np1 /sys/subsystem/net/devices/eni99np1 E: TAGS=:systemd: ➜ ~ echo "3" /sys/class/net/eni99np1/device/sriov_numvfs 3 /sys/class/net/eni99np1/device/sriov_numvfs ➜ ~ udevadm info -w10s /sys/devices/netdevsim99/net/eni99np1 P: /devices/netdevsim99/net/eni99np1 L: 0 E: DEVPATH=/devices/netdevsim99/net/eni99np1 E: SUBSYSTEM=net E: INTERFACE=eni99np1 E: IFINDEX=9 E: USEC_INITIALIZED=53327702503 E: ID_NET_NAMING_SCHEME=v245 E: ID_NET_NAME_PATH=eni99np1 E: ID_NET_DRIVER=netdevsim E: ID_NET_LINK_FILE=/usr/lib/systemd/network/99-default.link E: ID_NET_NAME=eni99np1 E: SYSTEMD_ALIAS=/sys/subsystem/net/devices/eni99np1 /sys/subsystem/net/devices/eni99np1 E: TAGS=:systemd: ➜ ~ ip -d link show eni99np1 9: eni99np1: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether f6:e7:ab:f4:5b:0d brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65535 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 portname p1 switchid efa555440bc0d5e49db2492abc36907aa49440e0256787a78ab0288dc8bb00 ➜ ~ ➜ ~ echo "3" > /sys/class/net/eni99np1/device/sriov_numvfs ➜ ~ ip -d link show eni99np1 9: eni99np1: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether f6:e7:ab:f4:5b:0d brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65535 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 portname p1 switchid efa555440bc0d5e49db2492abc36907aa49440e0256787a78ab0288dc8bb00 vf 0 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off vf 1 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off vf 2 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off

I tested the same manually . It's ok

ssahani · 2020-06-22T11:33:07Z

This looks good. thanks @yuwata .

ssahani · 2020-06-22T11:35:29Z

test/test-network/systemd-networkd-tests.py

+        call('rmmod netdevsim', stderr=subprocess.STDOUT)
+        rc = call('modprobe netdevsim', stderr=subprocess.STDOUT)
+        if rc != 0:
+            return unittest.expectedFailure(func)


yes fedora does not support till now.

~ sus@esx-fedora ❯ sudo modprobe netdevsim modprobe: FATAL: Module netdevsim not found in directory /lib/modules/5.8.0-0.rc1.20200617git69119673bd50.1.fc33.x86_64

kernel-modules-internal-5.6.19-300.fc32.x86_64 provides netdevsim.

yuwata · 2020-06-22T13:45:27Z

@mrc0mmand Have you updated the Arch image for CentOS CIs? I have no idea about the CI failures.

ssahani · 2020-06-22T13:46:46Z

It seems some new parameters shows up in the route table.

mrc0mmand · 2020-06-22T13:49:53Z

@mrc0mmand Have you updated the Arch image for CentOS CIs? I have no idea about the CI failures.

Nope, the image update is stuck on #16199.

mrc0mmand · 2020-06-22T13:51:00Z

It seems some new parameters shows up in the route table.

Yeah,rt_trap appears in the routing table now, but not sure where it comes from.

yuwata · 2020-06-22T13:53:26Z

More seriously, now test_route_static does not configure the default gateway...

ssahani · 2020-06-22T15:29:11Z

It seems some new parameters shows up in the route table.

Yeah,rt_trap appears in the routing table now, but not sure where it comes from.

iproute update ?

mrc0mmand · 2020-06-22T15:47:38Z

I don't think so, there are several PRs which were built before and after this one, and all of them passed:

And the package set of the Arch image should be unchanged since Jun 11th.

ssahani · 2020-06-22T16:41:54Z

Please retrigger 2/3 times . If this happens again they we figure must be because of this PR and we need to do some cleanup.

mrc0mmand · 2020-06-22T18:18:00Z

It fails consistently with the exact same issue in both runs even after reschedule, whereas other (and more recent) PRs are fine.

src/network/networkd-sriov.c

man/systemd.network.xml

yuwata · 2020-06-24T04:01:14Z

Still I do not know why the rt_trap flag is added. So, I added a workaround in test_sysctl_disable_ipv6 to ignore the flag. Now all tests are green.

SR-IOV provides the ability to partition a single physical PCI resource into virtual PCI functions which can then be injected in to a VM. In the case of network VFs, SR-IOV improves north-south n etwork performance (that is, traffic with endpoints outside the host machine) by allowing traffic to bypass the host machine’s network stack.

This also adjusts test_sysctl_disable_ipv6(), as rt_trap flag may be added to routes on some environments.

yuwata · 2020-07-01T04:53:57Z

@keszybz Thank you for the review. Updated version is pushed. I hope all your points are addressed. PTAL.

keszybz · 2020-07-01T09:07:10Z

LGTM.

daiaji · 2022-01-23T18:23:08Z

How should I use it?

cat /etc/systemd/network/sr-iov.network 
[Match]
Name=enp7s0f2

[SR-IOV]
VirtualFunction=2

I created the relevant configuration file, but it doesn't seem to work.

yuwata · 2022-01-23T18:42:14Z

@daiaji Does the interface already has virtual functions? The [SR-IOV] section in .network file only configures existing virtual function. If the interface does not have it, then you need to add it by the following:

# echo 3 > /sys/class/net/enp7s0f2/device/sriov_numvfs

Note, recently I posted PR #21865 which adds the support of the same section into .link file, which creates relevant virtual function(s) on demand. If you'd like, please test the current git HEAD.

daiaji · 2022-01-24T02:24:54Z

@yuwata After compiling with this version of systemd, is there no need to manually use the sysfs user interface or add the kernel boot parameters to enable vf?

yuwata · 2022-01-24T10:46:55Z

I am not familiar with kernel boot params about SR-IOV. But, at least with the current git HEAD, it is not necessary to change sysfs attribute manually.

One note, as you may know, to make SR-IOV setting work, the interface needs to support the functionality.

daiaji · 2022-01-24T16:24:39Z

@yuwata

cat /etc/systemd/network/sr-iov.link   
[Match]
MACAddress=a0:36:9f:xx:xx:xx

[SR-IOV]
VirtualFunction=2

The link file does not appear to create VF.
Still need to use the command # echo 3 > /sys/class/net/enp7s0f2/device/sriov_numvfs
Or am I misunderstanding, systemd doesn't create vf for users?

yuwata · 2022-01-24T16:32:01Z

Did you reboot the system or run udevadm control --reload; udevadm trigger --action add /sys/class/net/enp7s0f2? If these do not work for you, then please open a new issue page with debugging logs.

daiaji · 2022-01-24T17:39:12Z

@yuwata I rebooted the system, and I confirmed that this Intel I350-T4 supports SR-IOV.
I will probably have more tests at this time tomorrow, so I should probably rest now.😀

yuwata · 2022-01-24T17:45:04Z

Thanks. Testing the feature is highly appreciated. I am waiting for your report in a new issue page.

daiaji · 2022-01-25T02:59:00Z

Until then, what logs do I need to provide? And how to access these logs.

yuwata · 2022-01-25T09:04:17Z

I noticed that your .link file shown in the above does not have number prefix. One of the matching .link file in the earliest in alphanumerical order is used, so, I guess still 99-default.link is applied to the interface. Please try to rename the file to e.g. 10-sr-iov.link.

And how to access these logs.

Do you use systemd-udevd from the current git HEAD with .link files? If so, create the following rules:

# /etc/udev/rules.d/00-debug-net.rules
SUBSYSTEM=="net", OPTIONS="log_level=debug"

Then, udevadm control --reload; udevadm trigger --action add /sys/class/net/enp7s0f2, and journalctl -b -u systemd-udevd.service shows the debugging logs. Try to export the logs with

journalctl -o short --no-hostname -b -u systemd-udevd.service > /tmp/udevd.log

and provide it.

Also, please provide udevadm info /sys/class/net/enp7s0f2.

Thank you!

daiaji · 2022-01-25T12:03:43Z

I noticed that your .link file shown in the above does not have number prefix. One of the matching .link file in the earliest in alphanumerical order is used, so, I guess still 99-default.link is applied to the interface. Please try to rename the file to e.g. 10-sr-iov.link.

OK, after changing the file name of the link file to 10-sr-iov.link, the VF is allocated normally.
But it seems that the interface name has changed to eth3, why is this?
and PF doesn't seem to be online after restarting the computer, how to make it online?

yuwata · 2022-01-25T15:35:35Z

OK, after changing the file name of the link file to 10-sr-iov.link, the VF is allocated normally.

Yay! Nice!

But it seems that the interface name has changed to eth3, why is this?

That's expected. As the .link file does not contain any naming (and MAC address) related settings. Copy the following lines from the 99-default.link:

[Link]
NamePolicy=keep kernel database onboard slot path
AlternativeNamesPolicy=database onboard slot path
MACAddressPolicy=persistent

and PF doesn't seem to be online after restarting the computer, how to make it online?

Set the link state to yes:

[SR-IOV]
VirtualFunction=2
LinkState=yes

For details, please see section about [SR-IOV] section in systemd.network(5) or systemd.link(5). There exists multiple settings in the section.

Note, if you want to online all VFs, then you need to specify multiple [SR-IOV] sections for each VF.

[SR-IOV]
VirtualFunction=0
LinkState=yes

[SR-IOV]
VirtualFunction=1
LinkState=yes

[SR-IOV]
VirtualFunction=2
LinkState=yes

daiaji · 2022-01-26T05:27:57Z

cat /etc/systemd/network/sr-iov.network
[Match]
Name=enp7s0f2

[Link]
ActivationPolicy=always-up

These configs worked for me.

yuwata · 2022-01-26T05:33:36Z

Yeah, that should also work. And from the logs in your recent issue about wait-online, it seems [SR-IOV] settings work fine for you.

daiaji · 2022-01-26T11:16:58Z

It seems to be more of a problem with the SR-IOV itself. I did not insert a network cable into the enp7s0f2 interface, so even if I use the VF as the network card of the VM, it seems that the network is still not available.

The ActivationPolicy=always-up parameter doesn't seem to help me.😅

But actually because SR-IOV and MACVLAN have similar characteristics (poor compatibility with IEEE 802.11), I can't access my AP, I have given up using SR-IOV and MACVLAN to build my soft router.

Will setting a VLAN help me?

yuwata · 2022-01-26T12:02:11Z

Hm, LinkState=yes doe not help you?

How about setting ConfigureWithoutCarrier=yes (probably with ActivationPolicy=always-up) in .network file for the VF?

Unfortunately, I do not have any SR-IOV supporting real interfaces, but only dummy virtual interface... So, I have almost no way to debug the issue...

yuwata · 2022-01-26T12:08:16Z

If they do not work, how about to also set ConfigureWithoutCarrier=yes (and ActivationPolicy=always-up) in PF?

yuwata · 2022-01-26T12:10:06Z

If you have any good way, then please let me know.

Thank you for your help of testing the [SR-IOV] feature and investigating the root cause of the wait-online issue!

daiaji · 2022-01-26T12:23:04Z

LinkState=yes seems to cause enp7s0f2 to not work directly.

daiaji · 2022-01-26T14:28:03Z

https://gist.github.com/daiaji/68df4ff15ba91edca3538ea8b4b65dc7

ssahani force-pushed the network-sr-iov branch 2 times, most recently from d85bf9b to f4cbeb6 Compare June 19, 2020 12:57

ssahani force-pushed the network-sr-iov branch from 431cef8 to a0b1d7d Compare June 19, 2020 16:43

ssahani changed the title ~~network: Introduce SR-IOV (Do#t Merge)~~ network: Introduce SR-IOV Jun 20, 2020

ssahani force-pushed the network-sr-iov branch 3 times, most recently from 6639924 to 36f6723 Compare June 22, 2020 02:49

yuwata added the network label Jun 22, 2020

yuwata force-pushed the network-sr-iov branch from 36f6723 to 0280de7 Compare June 22, 2020 10:50

ssahani commented Jun 22, 2020

View reviewed changes

mrc0mmand reviewed Jun 22, 2020

View reviewed changes

src/network/networkd-sriov.c Outdated Show resolved Hide resolved

mrc0mmand reviewed Jun 22, 2020

View reviewed changes

src/network/networkd-sriov.c Outdated Show resolved Hide resolved

mrc0mmand reviewed Jun 22, 2020

View reviewed changes

yuwata force-pushed the network-sr-iov branch 2 times, most recently from 0e88799 to 0bacad6 Compare June 23, 2020 11:37

ssahani and others added 4 commits July 1, 2020 13:51

network: add VLANProtocol= setting in [SR-IOV] section

e64b31c

network: add MACAddress= setting in [SR-IOV] section

c3a5025

test-network: add tests for SR-IOV

3d2c269

This also adjusts test_sysctl_disable_ipv6(), as rt_trap flag may be added to routes on some environments.

yuwata force-pushed the network-sr-iov branch from 28e2609 to 3d2c269 Compare July 1, 2020 04:52

keszybz added the good-to-merge/waiting-for-ci 👍 PR is good to merge, but CI hasn't passed at time of review. Please merge if you see CI has passed label Jul 1, 2020

yuwata merged commit 1447dc6 into systemd:master Jul 2, 2020

ssahani deleted the network-sr-iov branch July 2, 2020 11:53

network: Introduce SR-IOV #16219

network: Introduce SR-IOV #16219

Conversation

ssahani commented Jun 19, 2020 • edited

lgtm-com bot commented Jun 19, 2020

yuwata commented Jun 22, 2020

yuwata commented Jun 22, 2020

ssahani Jun 22, 2020

Choose a reason for hiding this comment

ssahani commented Jun 22, 2020

ssahani Jun 22, 2020

Choose a reason for hiding this comment

yuwata Jun 22, 2020

Choose a reason for hiding this comment

yuwata commented Jun 22, 2020

ssahani commented Jun 22, 2020

mrc0mmand commented Jun 22, 2020

mrc0mmand commented Jun 22, 2020

yuwata commented Jun 22, 2020

ssahani commented Jun 22, 2020 • edited

mrc0mmand commented Jun 22, 2020

ssahani commented Jun 22, 2020

mrc0mmand commented Jun 22, 2020

yuwata commented Jun 24, 2020

yuwata commented Jul 1, 2020

keszybz commented Jul 1, 2020

daiaji commented Jan 23, 2022 • edited

yuwata commented Jan 23, 2022

daiaji commented Jan 24, 2022 • edited

yuwata commented Jan 24, 2022

daiaji commented Jan 24, 2022

yuwata commented Jan 24, 2022 • edited

daiaji commented Jan 24, 2022

yuwata commented Jan 24, 2022 • edited

daiaji commented Jan 25, 2022

yuwata commented Jan 25, 2022

daiaji commented Jan 25, 2022 • edited

yuwata commented Jan 25, 2022

daiaji commented Jan 26, 2022

yuwata commented Jan 26, 2022

daiaji commented Jan 26, 2022 • edited

yuwata commented Jan 26, 2022 • edited

yuwata commented Jan 26, 2022

yuwata commented Jan 26, 2022

daiaji commented Jan 26, 2022

daiaji commented Jan 26, 2022

ssahani commented Jun 19, 2020 •

edited

ssahani commented Jun 22, 2020 •

edited

daiaji commented Jan 23, 2022 •

edited

daiaji commented Jan 24, 2022 •

edited

yuwata commented Jan 24, 2022 •

edited

yuwata commented Jan 24, 2022 •

edited

daiaji commented Jan 25, 2022 •

edited

daiaji commented Jan 26, 2022 •

edited

yuwata commented Jan 26, 2022 •

edited