Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Network cards firmware missing on 2765.2.0 #359

Closed
agiterman1 opened this issue Mar 7, 2021 · 8 comments · Fixed by flatcar-archive/coreos-overlay#887
Closed

Network cards firmware missing on 2765.2.0 #359

agiterman1 opened this issue Mar 7, 2021 · 8 comments · Fixed by flatcar-archive/coreos-overlay#887
Assignees
@margamanterola
Labels
channel/alpha Issue concerns the Alpha channel. channel/beta Issue concerns the Beta channel. channel/stable Issue concerns the Stable channel. kind/bug Something isn't working

Comments

@agiterman1
Copy link

Description

after auto upgrade in the stable channel
from: release-2605.12.0
to: release-2765.2.0

Impact

the network is not working anymore! (netwrok is down)

tried to ip set link up... and it failed with complain:

Direct fw load for bnx2 failed with error -2

had to rollback and stop the upgrade service.
@pothos
Copy link
Member

pothos commented Mar 7, 2021

Please provide more details on what happened. Is sounds like the interface is down? Have you tried sudo systemctl restart systemd-networkd? It seems the NIC you have needs a firmware loaded and an issue could be that this firmware has changed with the new update or similar. I could start to search making my own assumptions on your error but it would be helpful to have more information upfront (like dmesg, journalctl output, networkctl status INTERFACE etc).

@agiterman1
Copy link
Author

agiterman1 commented Mar 8, 2021 •
edited
Loading

Yes, after the auto upgrade, the network was down and could not take it up.

The sever is STANDARD HP G6
no special nics.

The sever located in a cold data center... I saw that was auto upgrade, the kernel has changed and saw the missing module,
So, made rollback and stopped the auto upgrade.

here are section from the failed boot log: ( and attached the full log )

Mar 07 10:10:42 localhost kernel: bnx2 0000:02:00.0 eth0: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found at mem f0000000, IRQ 34, node addr f4:ce:46:84:87:98
Mar 07 10:10:42 localhost kernel: bnx2 0000:02:00.1 eth1: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found at mem ee000000, IRQ 35, node addr f4:ce:46:84:87:9a
Mar 07 10:10:42 localhost kernel: SCSI subsystem initialized
Mar 07 10:10:42 localhost kernel: bnx2 0000:04:00.0 eth2: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found at mem f6000000, IRQ 36, node addr 18:a9:05:71:97:a0
Mar 07 10:10:42 localhost systemd-udevd[434]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Mar 07 10:10:42 localhost kernel: bnx2 0000:04:00.1 eth3: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found at mem f4000000, IRQ 37, node addr 18:a9:05:71:97:a2
Mar 07 10:10:42 localhost kernel: bnx2 0000:07:00.0 eth4: Broadcom NetXtreme II BCM5709 1000Base-T (C0) PCI Express found at mem fa000000, IRQ 38, node addr 18:a9:05:71:18:48

Mar 07 10:10:56 core245 systemd[1]: Starting Wait for Network to be Configured...
Mar 07 10:10:56 core245 kernel: bnx2 0000:07:00.1: Direct firmware load for bnx2/bnx2-mips-09-6.2.1b.fw failed with error -2
Mar 07 10:10:56 core245 kernel: ipmi_si dmi-ipmi-si.0: Removing SMBIOS-specified kcs state machine in favor of ACPI
Mar 07 10:10:56 core245 kernel: bnx2: Can't load firmware file "bnx2/bnx2-mips-09-6.2.1b.fw"
Mar 07 10:10:56 core245 kernel: ipmi_si: Adding ACPI-specified kcs state machine
Mar 07 10:10:56 core245 kernel: bnx2 0000:07:00.0: Direct firmware load for bnx2/bnx2-mips-09-6.2.1b.fw failed with error -2
Mar 07 10:10:56 core245 kernel: ipmi_si 0000:01:04.6: probing via PCI
Mar 07 10:10:56 core245 kernel: bnx2: Can't load firmware file "bnx2/bnx2-mips-09-6.2.1b.fw"
Mar 07 10:10:56 core245 kernel: bnx2 0000:04:00.1: Direct firmware load for bnx2/bnx2-mips-09-6.2.1b.fw failed with error -2
Mar 07 10:10:56 core245 kernel: ipmi_si 0000:01:04.6: Could not setup I/O space
Mar 07 10:10:56 core245 kernel: bnx2: Can't load firmware file "bnx2/bnx2-mips-09-6.2.1b.fw"
Mar 07 10:10:56 core245 kernel: bnx2 0000:04:00.0: Direct firmware load for bnx2/bnx2-mips-09-6.2.1b.fw failed with error -2
Mar 07 10:10:56 core245 kernel: ipmi_si 0000:01:04.6: [mem 0xedef0000-0xedef00ff] regsize 1 spacing 1 irq 21
Mar 07 10:10:56 core245 kernel: bnx2: Can't load firmware file "bnx2/bnx2-mips-09-6.2.1b.fw"
Mar 07 10:10:56 core245 kernel: bnx2 0000:02:00.1: Direct firmware load for bnx2/bnx2-mips-09-6.2.1b.fw failed with error -2
Mar 07 10:10:56 core245 kernel: ipmi_si: Adding PCI-specified kcs state machine
Mar 07 10:10:56 core245 kernel: ipmi_si: Trying PCI-specified kcs state machine at mem address 0xedef0000, slave address 0x0, irq 21
Mar 07 10:10:56 core245 kernel: bnx2: Can't load firmware file "bnx2/bnx2-mips-09-6.2.1b.fw"
Mar 07 10:10:56 core245 kernel: bnx2 0000:02:00.0: Direct firmware load for bnx2/bnx2-mips-09-6.2.1b.fw failed with error -2
Mar 07 10:10:56 core245 systemd-networkd[793]: ens2f1: Could not bring up interface: No such file or directory
Mar 07 10:10:56 core245 systemd-networkd[793]: ens2f0: Could not bring up interface: No such file or directory
Mar 07 10:10:56 core245 systemd-networkd[793]: ens1f1: Could not bring up interface: No such file or directory
Mar 07 10:10:56 core245 systemd-networkd[793]: ens1f0: Could not bring up interface: No such file or directory
Mar 07 10:10:56 core245 systemd-networkd[793]: enp2s0f1: Could not bring up interface: No such file or directory
Mar 07 10:10:56 core245 systemd-networkd[793]: enp2s0f0: Could not bring up interface: No such file or directory
Mar 07 10:10:56 core245 kernel: bnx2: Can't load firmware file "bnx2/bnx2-mips-09-6.2.1b.fw"
M

boot.log

@agiterman1
Copy link
Author

I have found same machine, HP G6
made fresh install.

same problem.

attached screen shot
20210308_103953

@margamanterola margamanterola changed the title network breaked after auto upgrade to release-2765.2.0 Network cards firmware missing on 2765.2.0 Mar 8, 2021
@margamanterola
Copy link
Contributor

Thanks for the screenshot, that was really helpful in understanding the problem.

For searching purposes, the error message shown in the image is: bnx2: Can't load firmware file "bnx2/bnx2-mips-06-6.2.3.fw"

Unfortunately, this is not the only missing firmware in this image :(. I found that the following 64 files used to be shipped until 2605.12.0 and are now gone:

/usr/lib64/firmware/acenic/tg1.bin
/usr/lib64/firmware/acenic/tg2.bin
/usr/lib64/firmware/bnx2/bnx2-mips-06-6.2.3.fw
/usr/lib64/firmware/bnx2/bnx2-mips-09-6.2.1b.fw
/usr/lib64/firmware/bnx2/bnx2-rv2p-06-6.0.15.fw
/usr/lib64/firmware/bnx2/bnx2-rv2p-09-6.0.17.fw
/usr/lib64/firmware/bnx2/bnx2-rv2p-09ax-6.0.17.fw
/usr/lib64/firmware/bnx2x/bnx2x-e1-7.13.11.0.fw
/usr/lib64/firmware/bnx2x/bnx2x-e1h-7.13.11.0.fw
/usr/lib64/firmware/bnx2x/bnx2x-e2-7.13.11.0.fw
/usr/lib64/firmware/ct2fw-3.2.5.1.bin
/usr/lib64/firmware/ctfw-3.2.5.1.bin
/usr/lib64/firmware/cxgb3/ael2005_opt_edc.bin
/usr/lib64/firmware/cxgb3/ael2005_twx_edc.bin
/usr/lib64/firmware/cxgb3/ael2020_twx_edc.bin
/usr/lib64/firmware/cxgb3/t3b_psram-1.1.0.bin
/usr/lib64/firmware/cxgb3/t3c_psram-1.1.0.bin
/usr/lib64/firmware/cxgb3/t3fw-7.12.0.bin
/usr/lib64/firmware/cxgb4/t4fw-1.24.17.0.bin
/usr/lib64/firmware/cxgb4/t4fw.bin
/usr/lib64/firmware/cxgb4/t5fw-1.24.17.0.bin
/usr/lib64/firmware/cxgb4/t5fw.bin
/usr/lib64/firmware/cxgb4/t6fw-1.24.17.0.bin
/usr/lib64/firmware/cxgb4/t6fw.bin
/usr/lib64/firmware/e100/d101m_ucode.bin
/usr/lib64/firmware/e100/d101s_ucode.bin
/usr/lib64/firmware/e100/d102e_ucode.bin
/usr/lib64/firmware/isci/isci_firmware.bin
/usr/lib64/firmware/mellanox/mlxsw_spectrum-13.2000.1886.mfa2
/usr/lib64/firmware/myri10ge_ethp_z8e.dat
/usr/lib64/firmware/myri10ge_eth_z8e.dat
/usr/lib64/firmware/myri10ge_rss_ethp_z8e.dat
/usr/lib64/firmware/myri10ge_rss_eth_z8e.dat
/usr/lib64/firmware/phanfw.bin
/usr/lib64/firmware/qed/qed_init_values_zipped-8.37.7.0.bin
/usr/lib64/firmware/ql2100_fw.bin
/usr/lib64/firmware/ql2200_fw.bin
/usr/lib64/firmware/ql2300_fw.bin
/usr/lib64/firmware/ql2322_fw.bin
/usr/lib64/firmware/ql2400_fw.bin
/usr/lib64/firmware/ql2500_fw.bin
/usr/lib64/firmware/rtl_nic/rtl8105e-1.fw
/usr/lib64/firmware/rtl_nic/rtl8106e-1.fw
/usr/lib64/firmware/rtl_nic/rtl8106e-2.fw
/usr/lib64/firmware/rtl_nic/rtl8107e-1.fw
/usr/lib64/firmware/rtl_nic/rtl8107e-2.fw
/usr/lib64/firmware/rtl_nic/rtl8125a-3.fw
/usr/lib64/firmware/rtl_nic/rtl8168d-1.fw
/usr/lib64/firmware/rtl_nic/rtl8168d-2.fw
/usr/lib64/firmware/rtl_nic/rtl8168e-1.fw
/usr/lib64/firmware/rtl_nic/rtl8168e-2.fw
/usr/lib64/firmware/rtl_nic/rtl8168e-3.fw
/usr/lib64/firmware/rtl_nic/rtl8168f-1.fw
/usr/lib64/firmware/rtl_nic/rtl8168f-2.fw
/usr/lib64/firmware/rtl_nic/rtl8168g-2.fw
/usr/lib64/firmware/rtl_nic/rtl8168g-3.fw
/usr/lib64/firmware/rtl_nic/rtl8168h-1.fw
/usr/lib64/firmware/rtl_nic/rtl8168h-2.fw
/usr/lib64/firmware/rtl_nic/rtl8402-1.fw
/usr/lib64/firmware/rtl_nic/rtl8411-1.fw
/usr/lib64/firmware/rtl_nic/rtl8411-2.fw
/usr/lib64/firmware/tigon/tg3.bin
/usr/lib64/firmware/tigon/tg3_tso5.bin
/usr/lib64/firmware/tigon/tg3_tso.bin

The code that bundles the firmware and decides what gets shipped and what doesn't get shipped, is here. This code lists the kernel module files that end in .ko and checks for the corresponding firmware, anything that doesn't get detected in this scan gets pruned in the next step.

The problem is caused by how the find command is issued. In 2605.12.0 we had:

core@localhost ~ $ ls -l /lib/modules/5.4.92-flatcar/kernel/drivers/net/ethernet/broadcom/
total 676
-rw-r--r--. 1 root root  74346 Jan 27 16:44 b44.ko
-rw-r--r--. 1 root root 147770 Jan 27 16:44 bnx2.ko
drwxr-xr-x. 2 root root   4096 Jan 27 17:39 bnx2x
drwxr-xr-x. 2 root root   4096 Jan 27 17:39 bnxt
-rw-r--r--. 1 root root 106186 Jan 27 16:44 cnic.ko
-rw-r--r--. 1 root root 305074 Jan 27 16:44 tg3.ko

But in 2765.2.0 we have:

core@localhost ~ $ ls -l /lib/modules/5.10.19-flatcar/kernel/drivers/net/ethernet/broadcom/
total 196
-rw-r--r--. 1 root root 15400 Mar  2 18:35 b44.ko.xz
-rw-r--r--. 1 root root 38616 Mar  2 18:35 bnx2.ko.xz
drwxr-xr-x. 2 root root  4096 Mar  2 19:29 bnx2x
drwxr-xr-x. 2 root root  4096 Mar  2 19:29 bnxt
-rw-r--r--. 1 root root 27404 Mar  2 18:35 cnic.ko.xz
-rw-r--r--. 1 root root 75600 Mar  2 18:35 tg3.ko.xz

The kernel module files are now compressed and thus they don't get listed with that find command. We need to change the code to look for either .ko or .ko.xz.

@margamanterola margamanterola self-assigned this Mar 8, 2021
@margamanterola margamanterola added channel/alpha Issue concerns the Alpha channel. channel/beta Issue concerns the Beta channel. channel/stable Issue concerns the Stable channel. kind/bug Something isn't working labels Mar 8, 2021
margamanterola pushed a commit to flatcar-archive/coreos-overlay that referenced this issue Mar 8, 2021
sys-kernel/coreos-firmware: include compressed modules
f96284b 
When listing kernel modules to decide which firmware should be shipped
together with the image, we need to now list both compressed and
uncompressed module.

Fixes: flatcar/Flatcar#359
@margamanterola margamanterola mentioned this issue Mar 8, 2021
Merged
margamanterola pushed a commit to flatcar-archive/coreos-overlay that referenced this issue Mar 8, 2021
sys-kernel/coreos-firmware: include compressed modules
817ec4e 
When listing kernel modules to decide which firmware should be shipped
together with the image, we need to now list both compressed and
uncompressed module.

Fixes: flatcar/Flatcar#359
margamanterola pushed a commit to flatcar-archive/coreos-overlay that referenced this issue Mar 8, 2021
sys-kernel/coreos-firmware: include compressed modules
c3d71e5 
When listing kernel modules to decide which firmware should be shipped
together with the image, we need to now list both compressed and
uncompressed module.

Fixes: flatcar/Flatcar#359
@margamanterola
Copy link
Contributor

The fix is in, this will be released with the next set of patched versions. Likely to happen later this week.

@AndreiCherniaev
Copy link

firmware d102e_ucode.bin for driver e100 missing

How I can fix it? Method A. Do steps:
0) My board's driver mount to my laptop on /media/a/rootfs

  1. cd /media/a/rootfs/lib/firmware/
  2. mkdir e100
  3. copy d102e_ucode.bin to your board's drive
    sudo cp /home/a/myb/d102e_ucode.bin /media/a/rootfs/lib/firmware/
  4. Now it should be work.

Or use alternative method B. Just set this:

Screenshot from 2022-11-15 13-36-17

@pothos
Copy link
Member

pothos commented Nov 15, 2022

@AndreiCherniaev Are you talking about Flatcar Container Linux?

@AndreiCherniaev
Copy link

@AndreiCherniaev Are you talking about Flatcar Container Linux?

no, I am about buildroot. Sorry.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@margamanterola margamanterola

Labels
channel/alpha Issue concerns the Alpha channel. channel/beta Issue concerns the Beta channel. channel/stable Issue concerns the Stable channel. kind/bug Something isn't working
Projects
None yet
Milestone
No milestone
4 participants
@pothos @agiterman1 @AndreiCherniaev @margamanterola