systemd predictable device naming with Intel X552/X557-AT #1390

Closed
major opened this Issue Sep 25, 2015 · 26 comments

Projects

None yet

4 participants

@major
Contributor
major commented Sep 25, 2015

I've just purchased a Supermicro 5028D-TN4T and it has two onboard network devices:

  • Intel I350 (two ports)
  • Intel X552/X557-AT 10GBASE-T (two ports)

The I350 shows up as eno1 and eno2, which is totally reasonable since they're onboard NICs. However, the dual-port X552/X557 card is onboard as well and shows up as eth0 and eth1.

lcpci output:

# lspci | grep Eth
03:00.0 Ethernet controller: Intel Corporation Ethernet Connection X552/X557-AT 10GBASE-T
03:00.1 Ethernet controller: Intel Corporation Ethernet Connection X552/X557-AT 10GBASE-T
05:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
05:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)

udevadm info -e output

P: /devices/pci0000:00/0000:00:02.2/0000:03:00.0
E: DEVPATH=/devices/pci0000:00/0000:00:02.2/0000:03:00.0
E: DRIVER=ixgbe
E: ID_MODEL_FROM_DATABASE=Ethernet Connection X552/X557-AT 10GBASE-T
E: ID_PCI_CLASS_FROM_DATABASE=Network controller
E: ID_PCI_SUBCLASS_FROM_DATABASE=Ethernet controller
E: ID_VENDOR_FROM_DATABASE=Intel Corporation
E: MODALIAS=pci:v00008086d000015ADsv000015D9sd000015ADbc02sc00i00
E: PCI_CLASS=20000
E: PCI_ID=8086:15AD
E: PCI_SLOT_NAME=0000:03:00.0
E: PCI_SUBSYS_ID=15D9:15AD
E: SUBSYSTEM=pci
E: USEC_INITIALIZED=3732889

P: /devices/pci0000:00/0000:00:02.2/0000:03:00.0/net/eth0
E: DEVPATH=/devices/pci0000:00/0000:00:02.2/0000:03:00.0/net/eth0
E: ID_BUS=pci
E: ID_MODEL_FROM_DATABASE=Ethernet Connection X552/X557-AT 10GBASE-T
E: ID_MODEL_ID=0x15ad
E: ID_NET_DRIVER=ixgbe
E: ID_NET_LINK_FILE=/usr/lib/systemd/network/99-default.link
E: ID_NET_NAME=eno1
E: ID_NET_NAME_MAC=enx0cc47a7591c8
E: ID_NET_NAME_ONBOARD=eno1
E: ID_NET_NAME_PATH=enp3s0f0
E: ID_OUI_FROM_DATABASE=Super Micro Computer, Inc.
E: ID_PATH=pci-0000:03:00.0
E: ID_PATH_TAG=pci-0000_03_00_0
E: ID_PCI_CLASS_FROM_DATABASE=Network controller
E: ID_PCI_SUBCLASS_FROM_DATABASE=Ethernet controller
E: ID_VENDOR_FROM_DATABASE=Intel Corporation
E: ID_VENDOR_ID=0x8086
E: IFINDEX=4
E: INTERFACE=eth0
E: SUBSYSTEM=net
E: SYSTEMD_ALIAS=/sys/subsystem/net/devices/eno1
E: TAGS=:systemd:
E: USEC_INITIALIZED=7449982

P: /devices/pci0000:00/0000:00:02.2/0000:03:00.1
E: DEVPATH=/devices/pci0000:00/0000:00:02.2/0000:03:00.1
E: DRIVER=ixgbe
E: ID_MODEL_FROM_DATABASE=Ethernet Connection X552/X557-AT 10GBASE-T
E: ID_PCI_CLASS_FROM_DATABASE=Network controller
E: ID_PCI_SUBCLASS_FROM_DATABASE=Ethernet controller
E: ID_VENDOR_FROM_DATABASE=Intel Corporation
E: MODALIAS=pci:v00008086d000015ADsv000015D9sd000015ADbc02sc00i00
E: PCI_CLASS=20000
E: PCI_ID=8086:15AD
E: PCI_SLOT_NAME=0000:03:00.1
E: PCI_SUBSYS_ID=15D9:15AD
E: SUBSYSTEM=pci
E: USEC_INITIALIZED=3732891

P: /devices/pci0000:00/0000:00:02.2/0000:03:00.1/net/eth1
E: DEVPATH=/devices/pci0000:00/0000:00:02.2/0000:03:00.1/net/eth1
E: ID_BUS=pci
E: ID_MODEL_FROM_DATABASE=Ethernet Connection X552/X557-AT 10GBASE-T
E: ID_MODEL_ID=0x15ad
E: ID_NET_DRIVER=ixgbe
E: ID_NET_LINK_FILE=/usr/lib/systemd/network/99-default.link
E: ID_NET_NAME=eno2
E: ID_NET_NAME_MAC=enx0cc47a7591c9
E: ID_NET_NAME_ONBOARD=eno2
E: ID_NET_NAME_PATH=enp3s0f1
E: ID_OUI_FROM_DATABASE=Super Micro Computer, Inc.
E: ID_PATH=pci-0000:03:00.1
E: ID_PATH_TAG=pci-0000_03_00_1
E: ID_PCI_CLASS_FROM_DATABASE=Network controller
E: ID_PCI_SUBCLASS_FROM_DATABASE=Ethernet controller
E: ID_VENDOR_FROM_DATABASE=Intel Corporation
E: ID_VENDOR_ID=0x8086
E: IFINDEX=5
E: INTERFACE=eth1
E: SUBSYSTEM=net
E: SYSTEMD_ALIAS=/sys/subsystem/net/devices/eno2
E: TAGS=:systemd:
E: USEC_INITIALIZED=7952695

If I'm reading the code in systemd correctly, ID_NET_NAME_ONBOARD will be chosen for the name, but that would clash with the I350 card that already has eno1 and eno2. I assume that's why the X552/X557 card shows up as eth0 and eth1. However, should the X552/X557 show up as enp3s0f0 and enp3s0f1 as a fallback?

Thanks!

@major
Contributor
major commented Sep 25, 2015

Currently running: systemd-222-4.fc23.x86_64 on 4.2.1-300.fc23.x86_64

@arvidjaar
Contributor

However, should the X552/X557 show up as enp3s0f0 and enp3s0f1 as a fallback?

My guess is that udev tries to rename them but fails which explains why they stay at original names. Do you see any errors in logs? But interesting is also SYSTEMD_ALIAS which is completely wrong at this point.

@kaysievers
Member

Looks like a "broken" firmware, that tries to assign the same onboard index to multiple cards.

@kaysievers
Member

But interesting is also SYSTEMD_ALIAS which is completely wrong at this point.
Seems, the rule needs to be fixed to use:
ENV{SYSTEMD_ALIAS}+="/sys/subsystem/net/devices/%k"
which is the resulting name, not the intended name during rule processing.

@kaysievers
Member

Let's have a look what the kernel exports to us. Does this work. (Just typing here, I might have messed it up)
$ grep . /sys/class/net/*/device/{index,label,acpi_index}

@arvidjaar
Contributor

Seems, the rule needs to be fixed to use:
ENV{SYSTEMD_ALIAS}+="/sys/subsystem/net/devices/%k"

Won't it make alias wrong if rename was successful? We really need to use final interface name after rename was attempted, and I do not think current udev rules allow us to express it.

@kaysievers
Member

Won't it make alias wrong if rename was successful? We really need to use final interface name after rename was attempted, and I do not think current udev rules allow us to express it.

$kernel, %k will be changed with the rename.

@arvidjaar
Contributor

$kernel, %k will be changed with the rename

As far as I can tell rules are applied first then interface renamed. So while KERNEL will be updated, old values was already used in susbstitution.

                udev_rules_apply_to_event(rules, event,
                                          timeout_usec, timeout_warn_usec,
                                          properties_list);

                /* rename a new network interface, if needed */

Actually when testing I got both aliases - the first one from ADD event for original kernel name, the second one for MOVE event during interface rename.

@kaysievers
Member

As far as I can tell rules are applied first then interface renamed. So while KERNEL will be updated, old values was already used in susbstitution.

Ah, right. That should only work with RUN, which is substituted later.

@major
Contributor
major commented Sep 28, 2015

Wow, thanks for the interest in the report. ;)

@kaysievers Here's the output you asked for:

# grep . /sys/class/net/*/device/{index,label,acpi_index}
/sys/class/net/eno1/device/index:1
/sys/class/net/eno2/device/index:2
/sys/class/net/eth0/device/index:1
/sys/class/net/eth1/device/index:2
/sys/class/net/eno1/device/label: Intel i350 Ethernet #1
/sys/class/net/eno2/device/label: Intel i350 Ethernet #2
/sys/class/net/eth0/device/label: Intel X557-AT2 Ethernet #1
/sys/class/net/eth1/device/label: Intel X557-AT2 Ethernet #2
grep: /sys/class/net/*/device/acpi_index: No such file or directory
@arvidjaar
Contributor

/sys/class/net/eno1/device/index:1
/sys/class/net/eth0/device/index:1

Could you make output of dmidecode available?

@major
Contributor
major commented Sep 28, 2015
@arvidjaar
Contributor

Well, the firmware is indeed buggy. It is expected, that

Device Type Instance is a unique value (within a given onboard device type)

May be udev should indeed try each defined name policy in turn until one succeeds. But then again it cannot be expressed using current udev rules model.

@kaysievers
Member

I see no sane way for udev to work around such broken firmware. The current udev behavior looks acceptable, we better give up, instead of establishing cross-device dependency magic behavior.

You could report the bug to the vendor, maybe they care enough to fix it.

@arvidjaar
Contributor

The current udev behavior looks acceptable

Incorrect SYSTEMD_ALIAS still needs to be fixed.

@major
Contributor
major commented Sep 28, 2015

@kaysievers There is a new BIOS update for the machine, but I'm not sure if it will help. 😄 I'll give it a try.

@kaysievers
Member

Incorrect SYSTEMD_ALIAS still needs to be fixed.

I have no idea how.

@major
Contributor
major commented Sep 28, 2015

Is there a way for me to set custom names for the interfaces or provide hints to udev by using SYSTEMD_ALIAS? That might be a workaround for me in the meantime.

@kaysievers
Member

Sure, .link files can do all that:
http://www.freedesktop.org/software/systemd/man/systemd.link.html

You can also disable the "onboard naming" globally, if you can live with the physical names for all devices.

@major
Contributor
major commented Sep 28, 2015

Fantastic. I'll give that a try.

@major
Contributor
major commented Sep 28, 2015

Just did this:

# cat /etc/systemd/network/10gb.link 
[Match]
Driver=ixgbe

[Link]
NamePolicy=path

And now I have:

# ip link
6: enp3s0f0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 0c:c4:7a:75:91:c8 brd ff:ff:ff:ff:ff:ff
7: enp3s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 0c:c4:7a:75:91:c9 brd ff:ff:ff:ff:ff:ff

This is a good enough workaround for me for now. If I open a bug with Supermicro, what should I be asking them to do?

@kaysievers
Member

Ask them why they put DMI type 41 with duplicate index numbers (Type Instance) in the firmware.

Handle 0x0021, DMI type 41, 11 bytes
Onboard Device
Reference Designation: Intel i350 Ethernet #1
Type: Ethernet
Status: Enabled
Type Instance: 1
Bus Address: 0000:05:00.0

Handle 0x0022, DMI type 41, 11 bytes
Onboard Device
Reference Designation: Intel i350 Ethernet #2
Type: Ethernet
Status: Enabled
Type Instance: 2
Bus Address: 0000:05:00.1

Handle 0x0023, DMI type 41, 11 bytes
Onboard Device
Reference Designation: Intel X557-AT2 Ethernet #1
Type: Ethernet
Status: Enabled
Type Instance: 1
Bus Address: 0000:03:00.0

Handle 0x0024, DMI type 41, 11 bytes
Onboard Device
Reference Designation: Intel X557-AT2 Ethernet #2
Type: Ethernet
Status: Enabled
Type Instance: 2
Bus Address: 0000:03:00.1

@kaysievers
Member

http://www.dmtf.org/sites/default/files/standards/documents/DSP0134_2.7.1.pdf

7.42.2 Onboard Device Types

"Device Type Instance is a unique value (within a given onboard device type)
used to indicate the order the device is designated by the system.

For example, a system with two identical Ethernet NICs may designate one
NIC (with higher Bus/Device/Function=15/0/0) as the first onboard NIC (instance 1)
and the other NIC (with lower Bus/Device/Function =3/0/0) as the second
onboard NIC (instance 2)"

@major
Contributor
major commented Sep 29, 2015

Thanks, @kaysievers. I'll attempt to find out where to open a bug with the vendor on that. ;)

@poettering
Member

Hmm, wouldn't it make sense to fix the kernel do ignore broken DMI data? I mean, it sounds pretty straightforward to parse through the DMI data when first looking for the firmware interface index, and if the same index is assigned to two DMI objects ignore the whole thing? That way @major's system would simply get the non-firmware names and all should be good?

Of course, this is a kernel fix then. But it would fix the situation for everybody with borked DMI data like this.

@poettering
Member

So, I am pretty sure this should be fixed in the kernel: it should verify that the firmware data it passes to userspace is consistent. DMI data that assigns the same indexes twice is borked, and the kernel should check for that and suppress the data. It's not the job of userspace to distrust and revalidate the data the kernel already processed, the kernel needs to fix the up on its own.

I will close this now, please file a bug against the kernel!

@poettering poettering closed this Nov 3, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment