microcode-20200609 Release, at least 06-4e-03, hangs user's system #31
Comments
|
The differences between Does disabling SGX in the BIOS work around the hang using |
|
As requested by @vicamo, here is the output from It's on a Lenovo T460 laptop. |
Disabling SGX does not seems to help: SGX was initially on "Software controlled", so changed it to "Disabled", but laptop still hangs when booting the new initrd image. |
|
Thanks for testing. I think this needs Intel to investigate now |
|
Same thing on my system with a similar CPU. I had to uninstall the intel-microcode package to boot. here is the output of Arquitetura: x86_64
Modo(s) operacional da CPU: 32-bit, 64-bit
Ordem dos bytes: Little Endian
Address sizes: 39 bits physical, 48 bits virtual
CPU(s): 4
Lista de CPU(s) on-line: 0-3
Thread(s) per núcleo: 2
Núcleo(s) por soquete: 2
Soquete(s): 1
Nó(s) de NUMA: 1
ID de fornecedor: GenuineIntel
Família da CPU: 6
Modelo: 78
Nome do modelo: Intel(R) Core(TM) i5-6300U CPU @ 2.40GHz
Step: 3
CPU MHz: 500.006
CPU MHz máx.: 3000,0000
CPU MHz mín.: 400,0000
BogoMIPS: 4999.90
Virtualização: VT-x
cache de L1d: 64 KiB
cache de L1i: 64 KiB
cache de L2: 512 KiB
cache de L3: 3 MiB
CPU(s) de nó0 NUMA: 0-3
Vulnerability Itlb multihit: KVM: Mitigation: Split huge pages
Vulnerability L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
Vulnerability Mds: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Meltdown: Mitigation; PTI
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Full generic retpoline, IBPB conditional, IBRS_FW, STIBP conditional, RSB filling
Vulnerability Srbds: Vulnerable: No microcode
Vulnerability Tsx async abort: Mitigation; Clear CPU buffers; SMT vulnerable
Opções: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pn
i pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti ssbd ibrs ibpb sti
bp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp m
d_clear flush_l1d
`` |
|
On Wed, Jun 10, 2020 at 06:51:03AM -0700, Paulo J. S. Silva wrote:
Same thing on my system with a similar CPU. I had to uninstall the intel-microcode package to boot.
Have you tried using revision 0xd6 (from [1], for example) or 0xda
(from [2], for example) of the microcode?
[1] https://github.com/intel/Intel-Linux-Processor-Microcode-Data-Files/raw/437f382b1be4412b9d03e2bbdcda46d83d581242/intel-ucode/06-4e-03
[2] https://github.com/platomav/CPUMicrocodes/raw/master/Intel/cpu406E3_platC0_ver000000DA_2020-01-09_PRD_1527A73F.bin
|
|
I used the Ubuntu package intel-microcode (3.20200609.0ubuntu0.20.04.0). I don't have it in my system anymore to check out. From the description of the package in https://launchpad.net/ubuntu/+source/intel-microcode/3.20200609.0ubuntu0.20.04.0 I guess 0x00d6. Do you want me to reinstall it and check? |
|
On Wed, Jun 10, 2020 at 07:55:50AM -0700, Paulo J. S. Silva wrote:
I used the Ubuntu package intel-microcode (3.20200609.0ubuntu0.20.04.0). I don't have it in my system anymore to check out. From the description of the package in https://launchpad.net/ubuntu/+source/intel-microcode/3.20200609.0ubuntu0.20.04.0 I guess 0x00d6. Do you want me to reinstall it and check?
3.20200609.0ubuntu0.20.04.0 has revision 0xdc of 06-4e-03 microcode file,
3.20191115.1ubuntu3 has 0xd6.
|
|
So it is 0xdc (I found the reference in the middle of the description). As I said I can test what ever you want. Just point me to the file and instructions on how to apply it without an Ubuntu package. Thanks! |
|
Did you also update the Linux kernel? It would be nice to know if the regression is related to the microcode update itself, or to the combination of the two... |
|
It looks from my /var/log/apt/history.log that it also update the linux-image package to 5.4.0-37. This is the kernel I am using right now, it did not need to uninstall it. I only uninstalled intel-microcode and its dependencies (meta packages that install the latest kernel and friends). Both were unattended upgrades (because they were marked as a security update). |
|
Would it trouble you too much to test the intel-microcode package with an older kernel? With an old kernel, the microcode enables SRDBS mitigation in transparent mode, it is supposed to "just work" even if it might end up being enabled in a configuration that does not need it... This ensures the issue is not being caused by some bug in the new kernel code. |
|
If you do test with an older kernel, please look in /proc/cpuinfo to ensure you are indeed running microcode revision 0xdc, i.e. the new one for your processor. |
|
You mean try to use a mainline kernel? What version? I can use mainline PPA to do it easily. How do I install the Intel microcode without a deb package. Just point me to the docs, I can manage myself. |
|
On Wed, Jun 10, 2020 at 08:04:22AM -0700, Paulo J. S. Silva wrote:
So it is 0xdc (I found the reference in the middle of the description). As I said I can test what ever you want. Just point me to the file and instructions on how to apply it without an Ubuntu package.
Please try out [1] for 0xd6 revision. For 0xda revision, you can replace
"/lib/firmware/intel-ucode/06-4e-03" with [2] (don't forget to do backup
the original version) and re-generate initramfs with "update-initramfs -u"
command.
Note that one can always avoid early microcode loading during boot
by supplying "dis_ucode_ldr" kernel commandline option.
Thank you.
[1] https://launchpad.net/ubuntu/focal/amd64/intel-microcode/3.20191115.1ubuntu3
[2] https://github.com/platomav/CPUMicrocodes/raw/master/Intel/cpu406E3_platC0_ver000000DA_2020-01-09_PRD_1527A73F.bin
|
|
@pjssilva: no, please use any older Ubuntu kernel that works well in your machine for the purposes of this testing. |
|
You can also use Debian's intel-microcode packages, they should be compatible enough with Ubuntu. here: You can use the ones without any "~deb" on their name. |
|
@pjssilva FYI I'm preparing packages with a revert to version d6 for the 06-4e-03, they should be available for testing from https://launchpad.net/~sbeattie/+archive/ubuntu/lp1882890/ shortly. I would appreciate confirmation that they resolve the issue for you before publishing to the ubuntu archive. Also, the previous ubuntu kernel in 20.04 is 5.4.0-33, should be available to install via |
|
Also, as a datapoint, we did successfully test the combintation of the 0xdc microcode + a srbds aware kernel on a system with an i5-6200U before publishing the 20200609 update. |
|
I am in the middle of my testing spree. What I learned so far: 0xd6 seems to work always. As for 0xda: With 5.4.0-37-generic: it fails a lot, but not always. I managed to boot into gnome after booting first in rescue mode (to see if the right). Some time it boots OK, but most time not. It might be easier to boot after a full shutdown (instead of a reboot). With 5.4.0-33-generic: it seems to fail less, but it fails. Usually stuck after "loading initial ramdisk" (or somthing like this). I have the feeling that it does not fail if I do a full shutdown first. I will try that now (3 ful lshutdowns in a row). |
|
0xda did not pass the 3 cold boot test with 5.4.0-33-generic. So I can say that boot hangs with both 5.4.0-33 and 5.0.4-37 hot and cold boots. I don't see a pattern. @stevebeattie I will try the packages in your ppa now. I will let you know what I find out. |
|
Hi, Same issue here, and I confirm the fix/rollback proposed by @stevebeattie does solve the problem. My hardware: |
|
On Wed, Jun 10, 2020 at 10:23:38AM -0700, Paulo J. S. Silva wrote:
I am in the middle of my testing spree. What I learned so far:
0xd6 seems to work always.
As for 0xda:
With 5.4.0-37-generic: it fails a lot, but not always. I managed to boot into gnome after booting first in rescue mode (to see if the right). Some time it boots OK, but most time not. It might be easier to boot after a full shutdown (instead of a reboot).
With 5.4.0-33-generic: it seems to fail less, but it fails. Usually stuck after "loading initial ramdisk" (or somthing like this). I have the feeling that it does not fail if I do a full shutdown first. I will try that now (3 ful lshutdowns in a row).
Thank you very much for the testing!
|
|
@stevebeattie, as @Malizor I confirm that your package solves the problem. |
|
Well, looks like we (distros) will be forced to (re-)publish the security fix without 06-4e-03 :-( |
|
Just noticed 0x506e3 might have the same problem, but we got no reports (either positive or negative) thus far. |
|
Just wanted to leave a report that on an Asus UX305-CA with an Intel Core m3-6y30 (sig=0x406e3), using Arch Linux and after just updating intel-ucode package, the system will refuse to boot, I had to edit the grub entry at boot time to remove the intel-ucode line from initrd and it boots again. Made a full system upgrade running 5.6.15-arch1-1 kernel. This seems like a QA issue at Intel? |
|
Same issue here, boot freezes on Ubuntu 20.04/focal |
|
On Thu, Jun 11, 2020 at 09:49:51AM -0700, Mirek Ingr wrote:
Intel Core i7-8850H, sig=0x000906ea
This is a different CPUID, probably it should be reorted separately.
|
These reports show a failure later during boot and only with power unplugged. My failures were earlier (before the first line from Linux printed) and with the power cable plugged in. I didn't test without power cable plugged in. |
What is the version of the microcode present in your BIOS (i.e. which microcode is active when you do not have any intel-microcode packages installed )? |
|
Having the same boot problems for my Lenovo T460s here (i5-6200u). Without the intel-ucode it boots fine. Ucode package installed is intel-ucode 20201118-1 for Arch. I have microcode 0x74 active when no dedicated ucode package is loaded. |
This comment has been minimized.
This comment has been minimized.
|
@gerl1ng: this bug report is not about the i7-7700, so it would be best to hide/remove your comment about it: it just confuse things. As for your Lenovo T460 with a 0x406e3 processor on rev 0x74 that cannot be early-updated to the current microcode revision, that is a relevant data point. Thank you. |
I have the same problem on the current Arch Linux install on the same Laptop, my CPU is i7-6600U, microcode 0x74. Uninstalling intel-ucode restores bootability for now. What can I do? |
|
Same issue here, too. System doesn't boot running latest intel-ucode 20201118-1. Looks like the issue is still present. |
|
I also cannot boot when intel-ucode is installed. I am on a Dell Latitude 7400 with i7-8665U CPU and 5.10.70-arch-1. Are there any news on this problem? |
|
Problem Statement: Issue description: Workaround: |
|
@whpenner Unfortunately not the best suggestion for a workaround. We all know about the "update policy" of the manufacturers. On most of their products EOS is reached very soon (e.g. my ASUS received the last BIOS update in 2019). |
Thanks! That worked for me. |
|
intel core i9 also has a problem, I'm using latest BIOS version. More info about intel microcode bug here - https://bugs.launchpad.net/ubuntu/+source/intel-microcode/+bug/1911893. |
Please verify that your CPU has the signature corresponding to the issue discussed here (cannot find this info in your comment nor through the launchpad link). If that's a different signature, please open another dedicated issue. @whpenner: nice finding. Can you give a source? |
Please find all info here - https://forums.linuxmint.com/viewtopic.php?f=49&t=339625&sid=bdf869985cc73de93fc77e1ce330aa29 |
That core i9 is model 158, completely unrelated to this issue. |
@whpenner: thanks, however we do need some extra information to act upon:
If the answer to (1) above is "yes, now that Intel know where the problem is, there is a way to avoid it and Intel will update the microcode to avoid the issue", then we (distros) can remove the problematic update meanwhile, without the need for kernel patches, etc... |
|
So a recommended fix would be to update BIOS if possible and then attempt to use the new microcode package? |
|
@dbischof90: yes. |
|
If a bios update is not available from the motherboard manufacturer, I would highly recommend looking into UBU and patch the microcode into a bios update directly... Good luck expecting support from motherboard manufacturers... Last bios update for my server board was 4 years ago at this point (Thanks SuperMicro) |
|
@naitoshedo thanks for pointing me in that direction. Now everything's fine: |
|
New revision 0xea of 06-4e-03/06-5e-03 microcode files has been published as part of microcode-20210608 release, it may be worth trying out. |
|
The |
|
The above should be resolved with MCU revision ids of 0xe2 or higher for CPUID 406e3 (06-4e-03) and 506e3 (06-5e-03). MCUs released yesterday included revision id 0xea for these products. |
|
As already stated, the issue was fixed on a previous MCU version for my hardware (see previous comments), this |
|
Like in #31 (comment) the latest release also fails in the same way (no change to the bios): |
|
@JanZerebecki what microcode revision is reported when the kernel is loaded with |
|
@jirireischig may I ask to check if 0xe2/0xea microcode revision helps you (in case you still have the problematic (<0x80) microcode revision applied by firmware/BIOS), by chance? |
|
@whpenner, could you confirm if the latest revision (0xea) is supposed to work around the problem stated in #31 (comment) ? It would be really nice to be able to ship the microcode update without worrying about severely outdated BIOSes that cannot be fixed due to the usual motherboard-vendor-won't-issue-updated-firmware + EOL issues. |
|
Yes, this should resolve the issue called out in #31. This MCU was updated to work with either the older MCU (older than revision ID 0x80) and the newer MCUs. I would not recommend an early load of an affected MCU and then load of the latest MCU. The results may be unpredictable. Updating from a BIOS loaded MCU to the latest MCU directly is the recommended method. |
|
Thank you. Most Linux distros do exactly that: update directly from BIOS to latest MCU (via an early update), that's why we request a reboot to apply the microcode updates. It is important data for anyone doing late loads, though. Thank you for this information |
and others
Per reports from https://bugs.launchpad.net/ubuntu/+source/intel-microcode/+bug/1882890,
sig 0x000406e3, pf_mask 0xc0, 2020-04-27, rev 0x00dc, size 104448hangs user's system in a similar way as #24 does but to different cpu. It's also reported the same file of a previous revisionmicrocode: sig=0x406e3, pf=0x80, revision=0xd6works just fine.The text was updated successfully, but these errors were encountered: