New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

systemd predictable device naming with Intel X552/X557-AT #1390

Closed
major opened this Issue Sep 25, 2015 · 26 comments

Comments

4 participants
@major
Contributor

major commented Sep 25, 2015

I've just purchased a Supermicro 5028D-TN4T and it has two onboard network devices:

  • Intel I350 (two ports)
  • Intel X552/X557-AT 10GBASE-T (two ports)

The I350 shows up as eno1 and eno2, which is totally reasonable since they're onboard NICs. However, the dual-port X552/X557 card is onboard as well and shows up as eth0 and eth1.

lcpci output:

# lspci | grep Eth
03:00.0 Ethernet controller: Intel Corporation Ethernet Connection X552/X557-AT 10GBASE-T
03:00.1 Ethernet controller: Intel Corporation Ethernet Connection X552/X557-AT 10GBASE-T
05:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
05:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)

udevadm info -e output

P: /devices/pci0000:00/0000:00:02.2/0000:03:00.0
E: DEVPATH=/devices/pci0000:00/0000:00:02.2/0000:03:00.0
E: DRIVER=ixgbe
E: ID_MODEL_FROM_DATABASE=Ethernet Connection X552/X557-AT 10GBASE-T
E: ID_PCI_CLASS_FROM_DATABASE=Network controller
E: ID_PCI_SUBCLASS_FROM_DATABASE=Ethernet controller
E: ID_VENDOR_FROM_DATABASE=Intel Corporation
E: MODALIAS=pci:v00008086d000015ADsv000015D9sd000015ADbc02sc00i00
E: PCI_CLASS=20000
E: PCI_ID=8086:15AD
E: PCI_SLOT_NAME=0000:03:00.0
E: PCI_SUBSYS_ID=15D9:15AD
E: SUBSYSTEM=pci
E: USEC_INITIALIZED=3732889

P: /devices/pci0000:00/0000:00:02.2/0000:03:00.0/net/eth0
E: DEVPATH=/devices/pci0000:00/0000:00:02.2/0000:03:00.0/net/eth0
E: ID_BUS=pci
E: ID_MODEL_FROM_DATABASE=Ethernet Connection X552/X557-AT 10GBASE-T
E: ID_MODEL_ID=0x15ad
E: ID_NET_DRIVER=ixgbe
E: ID_NET_LINK_FILE=/usr/lib/systemd/network/99-default.link
E: ID_NET_NAME=eno1
E: ID_NET_NAME_MAC=enx0cc47a7591c8
E: ID_NET_NAME_ONBOARD=eno1
E: ID_NET_NAME_PATH=enp3s0f0
E: ID_OUI_FROM_DATABASE=Super Micro Computer, Inc.
E: ID_PATH=pci-0000:03:00.0
E: ID_PATH_TAG=pci-0000_03_00_0
E: ID_PCI_CLASS_FROM_DATABASE=Network controller
E: ID_PCI_SUBCLASS_FROM_DATABASE=Ethernet controller
E: ID_VENDOR_FROM_DATABASE=Intel Corporation
E: ID_VENDOR_ID=0x8086
E: IFINDEX=4
E: INTERFACE=eth0
E: SUBSYSTEM=net
E: SYSTEMD_ALIAS=/sys/subsystem/net/devices/eno1
E: TAGS=:systemd:
E: USEC_INITIALIZED=7449982

P: /devices/pci0000:00/0000:00:02.2/0000:03:00.1
E: DEVPATH=/devices/pci0000:00/0000:00:02.2/0000:03:00.1
E: DRIVER=ixgbe
E: ID_MODEL_FROM_DATABASE=Ethernet Connection X552/X557-AT 10GBASE-T
E: ID_PCI_CLASS_FROM_DATABASE=Network controller
E: ID_PCI_SUBCLASS_FROM_DATABASE=Ethernet controller
E: ID_VENDOR_FROM_DATABASE=Intel Corporation
E: MODALIAS=pci:v00008086d000015ADsv000015D9sd000015ADbc02sc00i00
E: PCI_CLASS=20000
E: PCI_ID=8086:15AD
E: PCI_SLOT_NAME=0000:03:00.1
E: PCI_SUBSYS_ID=15D9:15AD
E: SUBSYSTEM=pci
E: USEC_INITIALIZED=3732891

P: /devices/pci0000:00/0000:00:02.2/0000:03:00.1/net/eth1
E: DEVPATH=/devices/pci0000:00/0000:00:02.2/0000:03:00.1/net/eth1
E: ID_BUS=pci
E: ID_MODEL_FROM_DATABASE=Ethernet Connection X552/X557-AT 10GBASE-T
E: ID_MODEL_ID=0x15ad
E: ID_NET_DRIVER=ixgbe
E: ID_NET_LINK_FILE=/usr/lib/systemd/network/99-default.link
E: ID_NET_NAME=eno2
E: ID_NET_NAME_MAC=enx0cc47a7591c9
E: ID_NET_NAME_ONBOARD=eno2
E: ID_NET_NAME_PATH=enp3s0f1
E: ID_OUI_FROM_DATABASE=Super Micro Computer, Inc.
E: ID_PATH=pci-0000:03:00.1
E: ID_PATH_TAG=pci-0000_03_00_1
E: ID_PCI_CLASS_FROM_DATABASE=Network controller
E: ID_PCI_SUBCLASS_FROM_DATABASE=Ethernet controller
E: ID_VENDOR_FROM_DATABASE=Intel Corporation
E: ID_VENDOR_ID=0x8086
E: IFINDEX=5
E: INTERFACE=eth1
E: SUBSYSTEM=net
E: SYSTEMD_ALIAS=/sys/subsystem/net/devices/eno2
E: TAGS=:systemd:
E: USEC_INITIALIZED=7952695

If I'm reading the code in systemd correctly, ID_NET_NAME_ONBOARD will be chosen for the name, but that would clash with the I350 card that already has eno1 and eno2. I assume that's why the X552/X557 card shows up as eth0 and eth1. However, should the X552/X557 show up as enp3s0f0 and enp3s0f1 as a fallback?

Thanks!

@major

This comment has been minimized.

Show comment
Hide comment
@major

major Sep 25, 2015

Contributor

Currently running: systemd-222-4.fc23.x86_64 on 4.2.1-300.fc23.x86_64

Contributor

major commented Sep 25, 2015

Currently running: systemd-222-4.fc23.x86_64 on 4.2.1-300.fc23.x86_64

@arvidjaar

This comment has been minimized.

Show comment
Hide comment
@arvidjaar

arvidjaar Sep 26, 2015

Contributor

However, should the X552/X557 show up as enp3s0f0 and enp3s0f1 as a fallback?

My guess is that udev tries to rename them but fails which explains why they stay at original names. Do you see any errors in logs? But interesting is also SYSTEMD_ALIAS which is completely wrong at this point.

Contributor

arvidjaar commented Sep 26, 2015

However, should the X552/X557 show up as enp3s0f0 and enp3s0f1 as a fallback?

My guess is that udev tries to rename them but fails which explains why they stay at original names. Do you see any errors in logs? But interesting is also SYSTEMD_ALIAS which is completely wrong at this point.

@kaysievers

This comment has been minimized.

Show comment
Hide comment
@kaysievers

kaysievers Sep 26, 2015

Contributor

Looks like a "broken" firmware, that tries to assign the same onboard index to multiple cards.

Contributor

kaysievers commented Sep 26, 2015

Looks like a "broken" firmware, that tries to assign the same onboard index to multiple cards.

@kaysievers

This comment has been minimized.

Show comment
Hide comment
@kaysievers

kaysievers Sep 26, 2015

Contributor

But interesting is also SYSTEMD_ALIAS which is completely wrong at this point.
Seems, the rule needs to be fixed to use:
ENV{SYSTEMD_ALIAS}+="/sys/subsystem/net/devices/%k"
which is the resulting name, not the intended name during rule processing.

Contributor

kaysievers commented Sep 26, 2015

But interesting is also SYSTEMD_ALIAS which is completely wrong at this point.
Seems, the rule needs to be fixed to use:
ENV{SYSTEMD_ALIAS}+="/sys/subsystem/net/devices/%k"
which is the resulting name, not the intended name during rule processing.

@kaysievers

This comment has been minimized.

Show comment
Hide comment
@kaysievers

kaysievers Sep 26, 2015

Contributor

Let's have a look what the kernel exports to us. Does this work. (Just typing here, I might have messed it up)
$ grep . /sys/class/net/*/device/{index,label,acpi_index}

Contributor

kaysievers commented Sep 26, 2015

Let's have a look what the kernel exports to us. Does this work. (Just typing here, I might have messed it up)
$ grep . /sys/class/net/*/device/{index,label,acpi_index}

@arvidjaar

This comment has been minimized.

Show comment
Hide comment
@arvidjaar

arvidjaar Sep 26, 2015

Contributor

Seems, the rule needs to be fixed to use:
ENV{SYSTEMD_ALIAS}+="/sys/subsystem/net/devices/%k"

Won't it make alias wrong if rename was successful? We really need to use final interface name after rename was attempted, and I do not think current udev rules allow us to express it.

Contributor

arvidjaar commented Sep 26, 2015

Seems, the rule needs to be fixed to use:
ENV{SYSTEMD_ALIAS}+="/sys/subsystem/net/devices/%k"

Won't it make alias wrong if rename was successful? We really need to use final interface name after rename was attempted, and I do not think current udev rules allow us to express it.

@kaysievers

This comment has been minimized.

Show comment
Hide comment
@kaysievers

kaysievers Sep 27, 2015

Contributor

Won't it make alias wrong if rename was successful? We really need to use final interface name after rename was attempted, and I do not think current udev rules allow us to express it.

$kernel, %k will be changed with the rename.

Contributor

kaysievers commented Sep 27, 2015

Won't it make alias wrong if rename was successful? We really need to use final interface name after rename was attempted, and I do not think current udev rules allow us to express it.

$kernel, %k will be changed with the rename.

@arvidjaar

This comment has been minimized.

Show comment
Hide comment
@arvidjaar

arvidjaar Sep 27, 2015

Contributor

$kernel, %k will be changed with the rename

As far as I can tell rules are applied first then interface renamed. So while KERNEL will be updated, old values was already used in susbstitution.

                udev_rules_apply_to_event(rules, event,
                                          timeout_usec, timeout_warn_usec,
                                          properties_list);

                /* rename a new network interface, if needed */

Actually when testing I got both aliases - the first one from ADD event for original kernel name, the second one for MOVE event during interface rename.

Contributor

arvidjaar commented Sep 27, 2015

$kernel, %k will be changed with the rename

As far as I can tell rules are applied first then interface renamed. So while KERNEL will be updated, old values was already used in susbstitution.

                udev_rules_apply_to_event(rules, event,
                                          timeout_usec, timeout_warn_usec,
                                          properties_list);

                /* rename a new network interface, if needed */

Actually when testing I got both aliases - the first one from ADD event for original kernel name, the second one for MOVE event during interface rename.

@kaysievers

This comment has been minimized.

Show comment
Hide comment
@kaysievers

kaysievers Sep 27, 2015

Contributor

As far as I can tell rules are applied first then interface renamed. So while KERNEL will be updated, old values was already used in susbstitution.

Ah, right. That should only work with RUN, which is substituted later.

Contributor

kaysievers commented Sep 27, 2015

As far as I can tell rules are applied first then interface renamed. So while KERNEL will be updated, old values was already used in susbstitution.

Ah, right. That should only work with RUN, which is substituted later.

@major

This comment has been minimized.

Show comment
Hide comment
@major

major Sep 28, 2015

Contributor

Wow, thanks for the interest in the report. ;)

@kaysievers Here's the output you asked for:

# grep . /sys/class/net/*/device/{index,label,acpi_index}
/sys/class/net/eno1/device/index:1
/sys/class/net/eno2/device/index:2
/sys/class/net/eth0/device/index:1
/sys/class/net/eth1/device/index:2
/sys/class/net/eno1/device/label: Intel i350 Ethernet #1
/sys/class/net/eno2/device/label: Intel i350 Ethernet #2
/sys/class/net/eth0/device/label: Intel X557-AT2 Ethernet #1
/sys/class/net/eth1/device/label: Intel X557-AT2 Ethernet #2
grep: /sys/class/net/*/device/acpi_index: No such file or directory
Contributor

major commented Sep 28, 2015

Wow, thanks for the interest in the report. ;)

@kaysievers Here's the output you asked for:

# grep . /sys/class/net/*/device/{index,label,acpi_index}
/sys/class/net/eno1/device/index:1
/sys/class/net/eno2/device/index:2
/sys/class/net/eth0/device/index:1
/sys/class/net/eth1/device/index:2
/sys/class/net/eno1/device/label: Intel i350 Ethernet #1
/sys/class/net/eno2/device/label: Intel i350 Ethernet #2
/sys/class/net/eth0/device/label: Intel X557-AT2 Ethernet #1
/sys/class/net/eth1/device/label: Intel X557-AT2 Ethernet #2
grep: /sys/class/net/*/device/acpi_index: No such file or directory
@arvidjaar

This comment has been minimized.

Show comment
Hide comment
@arvidjaar

arvidjaar Sep 28, 2015

Contributor

/sys/class/net/eno1/device/index:1
/sys/class/net/eth0/device/index:1

Could you make output of dmidecode available?

Contributor

arvidjaar commented Sep 28, 2015

/sys/class/net/eno1/device/index:1
/sys/class/net/eth0/device/index:1

Could you make output of dmidecode available?

@major

This comment has been minimized.

Show comment
Hide comment
@major
Contributor

major commented Sep 28, 2015

@arvidjaar

This comment has been minimized.

Show comment
Hide comment
@arvidjaar

arvidjaar Sep 28, 2015

Contributor

Well, the firmware is indeed buggy. It is expected, that

Device Type Instance is a unique value (within a given onboard device type)

May be udev should indeed try each defined name policy in turn until one succeeds. But then again it cannot be expressed using current udev rules model.

Contributor

arvidjaar commented Sep 28, 2015

Well, the firmware is indeed buggy. It is expected, that

Device Type Instance is a unique value (within a given onboard device type)

May be udev should indeed try each defined name policy in turn until one succeeds. But then again it cannot be expressed using current udev rules model.

@kaysievers

This comment has been minimized.

Show comment
Hide comment
@kaysievers

kaysievers Sep 28, 2015

Contributor

I see no sane way for udev to work around such broken firmware. The current udev behavior looks acceptable, we better give up, instead of establishing cross-device dependency magic behavior.

You could report the bug to the vendor, maybe they care enough to fix it.

Contributor

kaysievers commented Sep 28, 2015

I see no sane way for udev to work around such broken firmware. The current udev behavior looks acceptable, we better give up, instead of establishing cross-device dependency magic behavior.

You could report the bug to the vendor, maybe they care enough to fix it.

@arvidjaar

This comment has been minimized.

Show comment
Hide comment
@arvidjaar

arvidjaar Sep 28, 2015

Contributor

The current udev behavior looks acceptable

Incorrect SYSTEMD_ALIAS still needs to be fixed.

Contributor

arvidjaar commented Sep 28, 2015

The current udev behavior looks acceptable

Incorrect SYSTEMD_ALIAS still needs to be fixed.

@major

This comment has been minimized.

Show comment
Hide comment
@major

major Sep 28, 2015

Contributor

@kaysievers There is a new BIOS update for the machine, but I'm not sure if it will help. 😄 I'll give it a try.

Contributor

major commented Sep 28, 2015

@kaysievers There is a new BIOS update for the machine, but I'm not sure if it will help. 😄 I'll give it a try.

@kaysievers

This comment has been minimized.

Show comment
Hide comment
@kaysievers

kaysievers Sep 28, 2015

Contributor

Incorrect SYSTEMD_ALIAS still needs to be fixed.

I have no idea how.

Contributor

kaysievers commented Sep 28, 2015

Incorrect SYSTEMD_ALIAS still needs to be fixed.

I have no idea how.

@major

This comment has been minimized.

Show comment
Hide comment
@major

major Sep 28, 2015

Contributor

Is there a way for me to set custom names for the interfaces or provide hints to udev by using SYSTEMD_ALIAS? That might be a workaround for me in the meantime.

Contributor

major commented Sep 28, 2015

Is there a way for me to set custom names for the interfaces or provide hints to udev by using SYSTEMD_ALIAS? That might be a workaround for me in the meantime.

@kaysievers

This comment has been minimized.

Show comment
Hide comment
@kaysievers

kaysievers Sep 28, 2015

Contributor

Sure, .link files can do all that:
http://www.freedesktop.org/software/systemd/man/systemd.link.html

You can also disable the "onboard naming" globally, if you can live with the physical names for all devices.

Contributor

kaysievers commented Sep 28, 2015

Sure, .link files can do all that:
http://www.freedesktop.org/software/systemd/man/systemd.link.html

You can also disable the "onboard naming" globally, if you can live with the physical names for all devices.

@major

This comment has been minimized.

Show comment
Hide comment
@major

major Sep 28, 2015

Contributor

Fantastic. I'll give that a try.

Contributor

major commented Sep 28, 2015

Fantastic. I'll give that a try.

@major

This comment has been minimized.

Show comment
Hide comment
@major

major Sep 28, 2015

Contributor

Just did this:

# cat /etc/systemd/network/10gb.link 
[Match]
Driver=ixgbe

[Link]
NamePolicy=path

And now I have:

# ip link
6: enp3s0f0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 0c:c4:7a:75:91:c8 brd ff:ff:ff:ff:ff:ff
7: enp3s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 0c:c4:7a:75:91:c9 brd ff:ff:ff:ff:ff:ff

This is a good enough workaround for me for now. If I open a bug with Supermicro, what should I be asking them to do?

Contributor

major commented Sep 28, 2015

Just did this:

# cat /etc/systemd/network/10gb.link 
[Match]
Driver=ixgbe

[Link]
NamePolicy=path

And now I have:

# ip link
6: enp3s0f0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 0c:c4:7a:75:91:c8 brd ff:ff:ff:ff:ff:ff
7: enp3s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 0c:c4:7a:75:91:c9 brd ff:ff:ff:ff:ff:ff

This is a good enough workaround for me for now. If I open a bug with Supermicro, what should I be asking them to do?

@kaysievers

This comment has been minimized.

Show comment
Hide comment
@kaysievers

kaysievers Sep 29, 2015

Contributor

Ask them why they put DMI type 41 with duplicate index numbers (Type Instance) in the firmware.

Handle 0x0021, DMI type 41, 11 bytes
Onboard Device
Reference Designation: Intel i350 Ethernet #1
Type: Ethernet
Status: Enabled
Type Instance: 1
Bus Address: 0000:05:00.0

Handle 0x0022, DMI type 41, 11 bytes
Onboard Device
Reference Designation: Intel i350 Ethernet #2
Type: Ethernet
Status: Enabled
Type Instance: 2
Bus Address: 0000:05:00.1

Handle 0x0023, DMI type 41, 11 bytes
Onboard Device
Reference Designation: Intel X557-AT2 Ethernet #1
Type: Ethernet
Status: Enabled
Type Instance: 1
Bus Address: 0000:03:00.0

Handle 0x0024, DMI type 41, 11 bytes
Onboard Device
Reference Designation: Intel X557-AT2 Ethernet #2
Type: Ethernet
Status: Enabled
Type Instance: 2
Bus Address: 0000:03:00.1

Contributor

kaysievers commented Sep 29, 2015

Ask them why they put DMI type 41 with duplicate index numbers (Type Instance) in the firmware.

Handle 0x0021, DMI type 41, 11 bytes
Onboard Device
Reference Designation: Intel i350 Ethernet #1
Type: Ethernet
Status: Enabled
Type Instance: 1
Bus Address: 0000:05:00.0

Handle 0x0022, DMI type 41, 11 bytes
Onboard Device
Reference Designation: Intel i350 Ethernet #2
Type: Ethernet
Status: Enabled
Type Instance: 2
Bus Address: 0000:05:00.1

Handle 0x0023, DMI type 41, 11 bytes
Onboard Device
Reference Designation: Intel X557-AT2 Ethernet #1
Type: Ethernet
Status: Enabled
Type Instance: 1
Bus Address: 0000:03:00.0

Handle 0x0024, DMI type 41, 11 bytes
Onboard Device
Reference Designation: Intel X557-AT2 Ethernet #2
Type: Ethernet
Status: Enabled
Type Instance: 2
Bus Address: 0000:03:00.1

@kaysievers

This comment has been minimized.

Show comment
Hide comment
@kaysievers

kaysievers Sep 29, 2015

Contributor

http://www.dmtf.org/sites/default/files/standards/documents/DSP0134_2.7.1.pdf

7.42.2 Onboard Device Types

"Device Type Instance is a unique value (within a given onboard device type)
used to indicate the order the device is designated by the system.

For example, a system with two identical Ethernet NICs may designate one
NIC (with higher Bus/Device/Function=15/0/0) as the first onboard NIC (instance 1)
and the other NIC (with lower Bus/Device/Function =3/0/0) as the second
onboard NIC (instance 2)"

Contributor

kaysievers commented Sep 29, 2015

http://www.dmtf.org/sites/default/files/standards/documents/DSP0134_2.7.1.pdf

7.42.2 Onboard Device Types

"Device Type Instance is a unique value (within a given onboard device type)
used to indicate the order the device is designated by the system.

For example, a system with two identical Ethernet NICs may designate one
NIC (with higher Bus/Device/Function=15/0/0) as the first onboard NIC (instance 1)
and the other NIC (with lower Bus/Device/Function =3/0/0) as the second
onboard NIC (instance 2)"

@major

This comment has been minimized.

Show comment
Hide comment
@major

major Sep 29, 2015

Contributor

Thanks, @kaysievers. I'll attempt to find out where to open a bug with the vendor on that. ;)

Contributor

major commented Sep 29, 2015

Thanks, @kaysievers. I'll attempt to find out where to open a bug with the vendor on that. ;)

@poettering

This comment has been minimized.

Show comment
Hide comment
@poettering

poettering Oct 3, 2015

Member

Hmm, wouldn't it make sense to fix the kernel do ignore broken DMI data? I mean, it sounds pretty straightforward to parse through the DMI data when first looking for the firmware interface index, and if the same index is assigned to two DMI objects ignore the whole thing? That way @major's system would simply get the non-firmware names and all should be good?

Of course, this is a kernel fix then. But it would fix the situation for everybody with borked DMI data like this.

Member

poettering commented Oct 3, 2015

Hmm, wouldn't it make sense to fix the kernel do ignore broken DMI data? I mean, it sounds pretty straightforward to parse through the DMI data when first looking for the firmware interface index, and if the same index is assigned to two DMI objects ignore the whole thing? That way @major's system would simply get the non-firmware names and all should be good?

Of course, this is a kernel fix then. But it would fix the situation for everybody with borked DMI data like this.

@poettering

This comment has been minimized.

Show comment
Hide comment
@poettering

poettering Nov 3, 2015

Member

So, I am pretty sure this should be fixed in the kernel: it should verify that the firmware data it passes to userspace is consistent. DMI data that assigns the same indexes twice is borked, and the kernel should check for that and suppress the data. It's not the job of userspace to distrust and revalidate the data the kernel already processed, the kernel needs to fix the up on its own.

I will close this now, please file a bug against the kernel!

Member

poettering commented Nov 3, 2015

So, I am pretty sure this should be fixed in the kernel: it should verify that the firmware data it passes to userspace is consistent. DMI data that assigns the same indexes twice is borked, and the kernel should check for that and suppress the data. It's not the job of userspace to distrust and revalidate the data the kernel already processed, the kernel needs to fix the up on its own.

I will close this now, please file a bug against the kernel!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment