networkd: add SR-IOV support

[prometheanfire]

systemd version the issue has been seen with

debian buster backports - systemd 244 (244.3-1~bpo10+1)

Used distribution

Debian Buster

Expected behaviour you didn't see

parent interface of vlan not being brought up

Unexpected behaviour you saw

Apr 29 01:21:13 dev-virt-hype-1 systemd-networkd[3683]: enp4s0f0v1.4009: loaded vlan
Apr 29 01:21:13 dev-virt-hype-1 systemd-networkd[3683]: enp4s0f0v1: New device has no master, continuing without
Apr 29 01:21:13 dev-virt-hype-1 systemd-networkd[3683]: enp4s0f0v1: Flags change: +MULTICAST +BROADCAST
Apr 29 01:21:13 dev-virt-hype-1 systemd-networkd[3683]: enp4s0f0v1: Link 5 added
Apr 29 01:21:13 dev-virt-hype-1 systemd-networkd[3683]: enp4s0f0v1: udev initialized link
Apr 29 01:21:13 dev-virt-hype-1 systemd-networkd[3683]: enp4s0f0v1: State changed: pending -> initialized
Apr 29 01:21:13 dev-virt-hype-1 systemd-networkd[3683]: enp4s0f0v1: Saved original MTU: 1500
Apr 29 01:21:14 dev-virt-hype-1 systemd-networkd[3683]: enp4s0f0v1: Link state is up-to-date
Apr 29 01:21:14 dev-virt-hype-1 systemd-networkd[3683]: enp4s0f0v1: found matching network '/etc/systemd/network/enp4s0f0v1.network'
Apr 29 01:21:14 dev-virt-hype-1 systemd-networkd[3683]: enp4s0f0v1: IPv6 successfully enabled
Apr 29 01:21:14 dev-virt-hype-1 systemd-networkd[3683]: enp4s0f0v1: Setting address genmode for link
Apr 29 01:21:14 dev-virt-hype-1 systemd-networkd[3683]: enp4s0f0v1: State changed: initialized -> configuring
Apr 29 01:21:14 dev-virt-hype-1 systemd-networkd[3683]: enp4s0f0v1: Enslaving by 'enp4s0f0v1.4009'
Apr 29 01:21:14 dev-virt-hype-1 systemd-networkd[3683]: enp4s0f0v1.4009: Creating
Apr 29 01:21:14 dev-virt-hype-1 systemd-networkd[3683]: enp4s0f0v1: Could not join netdev: Input/output error
Apr 29 01:21:14 dev-virt-hype-1 systemd-networkd[3683]: enp4s0f0v1: Failed
Apr 29 01:21:14 dev-virt-hype-1 systemd-networkd[3683]: enp4s0f0v1: State changed: configuring -> failed

Steps to reproduce the problem

enp4s0f0v1.4009.netdev:[NetDev]
enp4s0f0v1.4009.netdev:Name=enp4s0f0v1.4009
enp4s0f0v1.4009.netdev:Kind=vlan
enp4s0f0v1.4009.netdev:[VLAN]
enp4s0f0v1.4009.netdev:Id=4009

enp4s0f0v1.4009.network:[Match]
enp4s0f0v1.4009.network:Name=enp4s0f0v1.4009
enp4s0f0v1.4009.network:[Network]

enp4s0f0v1.network:[Match]
enp4s0f0v1.network:Name=enp4s0f0v1
enp4s0f0v1.network:[Network]
enp4s0f0v1.network:VLAN=enp4s0f0v1.4009

If I set enp4s0f0v1 up manually, enp4s0f0v1.4009 is created as expected

[prometheanfire]

specifically this seems to be a problem of not being able to manage the sr-iov aspect of interfaces maybe that's interfering somehow.

when I set enp4s0f0 up before networkd is (re)started, enp4s0f0v1 is still not set up (pf needs to be up for vf to be up) for enp4s0f0v1.4009 to then be brought up.

the dependency is as follows
(pf) enp4s0f0 -> (vf) enp4s0f0v1 -> (vlan) enp4s0f0v1.4009

systemd is set up to manage enp4s0f0v1 and enp4s0f0v1.4009, enp4s0f0 is set up by a helper script with the following.

Before=network-pre.target
Wants=network-pre.target

by setting the helper script to also up the vf interfaces (all 63 of them...) networkd can set up the vlan.

[yuwata]

How do you configure the pf in the script?

[prometheanfire]

That aspect had to be manually managed (another customer oneshot systemd service that runs before networking is started as systemd does not support configuring sr-iov interfaces.

[ssahani]

We need to support sr-iov .

[yuwata]

@ssahani Are there any good reference to setup sr-iov? I am not familiar with sr-iov, and I do not have any device which support sr-iov.

[ssahani]

Well even I am not . We can simulate it via netdevsim. I think https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/virtualization_host_configuration_and_guest_installation_guide/chap-virtualization_host_configuration_and_guest_installation_guide-sr_iov

➜  ~ ip link add type netdevsim
Error: netdevsim: Please use: echo "[ID] [PORT_COUNT]" > /sys/bus/netdevsim/new_device.
➜  ~ echo "5 2" > /sys/bus/netdevsim/new_device
➜  ~ ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 00:0c:29:77:f8:a9 brd ff:ff:ff:ff:ff:ff
    altname enp2s1
3: veth-peer@veth99: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether fe:fc:82:9d:20:56 brd ff:ff:ff:ff:ff:ff
4: veth99@veth-peer: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 32:df:cb:9e:01:46 brd ff:ff:ff:ff:ff:ff
5: eni5np1: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 56:8e:c8:eb:ed:16 brd ff:ff:ff:ff:ff:ff
6: eni5np2: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 26:b7:21:bd:aa:2a brd ff:ff:ff:ff:ff:ff

➜  ~ echo 3 > /sys/class/net/eni5np2/device/sriov_numvfs
➜  ~ ip -d link show eni5np1
5: eni5np1: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 56:8e:c8:eb:ed:16 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65535 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 portname p1 switchid f20e1be4754893bfe43c8fd6ad1f7177369bc9140fdb9145fcf901f7114577
    vf 0     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 1     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off
    vf 2     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, link-state auto, trust off, query_rss off

[ssahani]

We already have netdevsim support in networkd. So I think we can work on it.

[yuwata]

Oh nice! Thanks.