Regression: intel-ucode 20191113 causes lockup on reboot

[ahesford]

On an Arch Linux installation, intel-ucode 20191113 causes a lockup when rebooting a Dell Precision 5820 workstation with an Intel Xeon W-2145 CPU. The system cold boots fine, but once the system is running, a reboot will cause a lockup when the kernel is reloaded. This is a regression from intel-ucode 20190918 and affects more than one bootloader and at least the linux, linux-zen and linux-lts kernels distributed by Arch. Because these three kernels are all affected and switching to the earlier intel-ucode package resolves the issue, I believe this is an upstream issue rather than a problem with the Arch package.

Additional info:

Affected version: 20191113
Last working version: 20190918

Bootloaders affected:

• systemd-boot (from systemd 243, Arch package systemd-243.78-2)
• rEFInd (version 0.11.3, Arch package refind-efi-0.11.3-1)

Note: rEFInd was not configured to apply the microcode patch at boot. Instead, the system was cold-booted from systemd-boot to apply the microcode patch, the boot manager was replaced with rEFInd, and the system was warm-rebooted using rEFInd. Thus, the reboot lockup does not appear to be caused by the act of loading the microcode, but instead causes the CPU to lock up after applying the microcode at least once and warm resetting.

Kernels affected:

• linux (5.3.11, Arch package linux-5.3.11.1-1)
• linux longterm (4.19.84, Arch package linux-lts-4.19.84-1)
• linux zen (5.3.11, Arch package linux-zen-5.3.11.1-1)

For systemd-boot, the loader entry is:
title Arch Linux
linux /vmlinuz-linux
initrd /intel-ucode.img
initrd /initramfs-linux.img
options root=UUID=[masked] rw
options consoleblank=600
options audit=0

For rEFInd, the loader entry was created automatically using refind-install and applied the same kernel arguments (root, consoleblank and audit) as the systemd loader.

Steps to reproduce:

1. Install Arch Linux and any of the kernels listed above (other kernels may be similarly affected).
2. Install package intel-ucode 20191113-1.
3. Configure systemd-boot to boot with a loader entry like that above, making sure to load the microcode during boot.
4. Cold boot the system; everything should boot as expected.
5. Invoke "shutdown -r now".
6. After systemd-boot selects the kernel to boot, the system should hang on the "SHA256 validated" message.
7. Forcibly power down the system.
8. [In my case, attempted to turn on the system at this point will cause the fans to spin, then the system to immediately shut itself down; powering on a second time will bring up the system as expected.]
9. Replace intel-ucode 20191113-1 with version 20190918-1 and confirm that the system boots and reboots as expected.

[hmh]

Can you tell us the output of "cat /proc/cpuinfo" ?

[esyr-rh]

And grep -r . /sys/devices/system/cpu/vulnerabilities as well, please?

[ahesford]

Below is the content of /proc/cpuinfo for the first virtual CPU. This is an eight-core model with hyperthreading, so this block repeats 15 more times. The only differences are the frequencies (which obviously jump around), the core ID (which matches the processor index), and the apicid and initial apicid fields (which have the same value for each processor: twice the processor index for indices 0 thorugh 7, and twice the processor index plus one for indices 8 through 15).

If you want all of the other blocks, please let me know.

processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 85
model name	: Intel(R) Xeon(R) W-2145 CPU @ 3.70GHz
stepping	: 4
microcode	: 0x2000064
cpu MHz		: 1450.713
cache size	: 11264 KB
physical id	: 0
siblings	: 16
core id		: 0
cpu cores	: 8
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 22
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req md_clear flush_l1d
bugs		: cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs taa itlb_multihit
bogomips	: 7402.02
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:

[ahesford]

Below are the contents of /sys/devices/system/cpu/vulnerabilities.

/sys/devices/system/cpu/vulnerabilities/spectre_v2:Mitigation: Full generic retpoline, IBPB: conditional, IBRS_FW, STIBP: conditional, RSB filling
/sys/devices/system/cpu/vulnerabilities/itlb_multihit:KVM: Mitigation: Split huge pages
/sys/devices/system/cpu/vulnerabilities/mds:Mitigation: Clear CPU buffers; SMT vulnerable
/sys/devices/system/cpu/vulnerabilities/l1tf:Mitigation: PTE Inversion; VMX: conditional cache flushes, SMT vulnerable
/sys/devices/system/cpu/vulnerabilities/spec_store_bypass:Mitigation: Speculative Store Bypass disabled via prctl and seccomp
/sys/devices/system/cpu/vulnerabilities/tsx_async_abort:Mitigation: Clear CPU buffers; SMT vulnerable
/sys/devices/system/cpu/vulnerabilities/spectre_v1:Mitigation: usercopy/swapgs barriers and __user pointer sanitization
/sys/devices/system/cpu/vulnerabilities/meltdown:Mitigation: PTI

[esyr-rh]

Hm, may I ask to supply "debug" parameter to the kernel's command line?

[ahesford]

Adding "debug" does not cause a change of behavior after a warm reboot. The system locks up after the systemd-boot "SHA256 validated" message without any additional information. The dmesg output after the first (cold) boot with debug enabled is attached.

How else may I help to isolate this issue?

dmesg.debug.log

[sclarkson]

I'm experiencing this as well.

System information

Mobo: ASUS WS X299 SAGE
CPU: 9920x and 9820x
OS: Ubuntu 18.04.3

I've tried Ubuntu's kernel 5.0.0-36-generic, as well as the mainline 5.3.11. Both exhibit the problem. I've also tried the latest BIOS from ASUS.

Confirmed that upgrading to 20191112 from 20190918 caused the issue.

System hangs when GRUB tries to load the kernel after either running reboot on the command line or pressing the reset button on the motherboard.

[esyr-rh]

It seems that there's a new release has just made available[1], may I ask to try it out?

[1] https://github.com/intel/Intel-Linux-Processor-Microcode-Data-Files/releases/tag/microcode-20191115

[hagar-dunor]

Gentoo user, same issue reported here on a Core i9-7920x. 20191115 doesn't seem to do any better.

[esyr-rh]

Right, there are no updates for 06-55-04 in microcode-20191115, so there's no point in testing 2019115, my apologies.

[hmh]

Looks server-die related (all reports are from HEDT parts or Xeon parts) ?

Kabylake desktop and mobile (0x806e9, 0x906e9) are not showing the reboot issue here.
CoffeeLake mobile (0x906ea) did now show any reboot issues, either.

[esyr-rh]

Looks server-die related (all reports are from HEDT parts or Xeon parts) ?

Both reports are against CPUID 0x50654 parts, yes.

Skylake mobile and desktop (0x806e9, 0x906e9) are not showing the reboot issue here.

FYI, Skylake mobile/desktop have CPUID 0x406e3 and 0x506e3, respectively.

[esyr-rh]

Gentoo user, same issue reported here on a Core i9-7920x. 20191115 doesn't seem to do any better.

May I ask to try out microcode-20190918 release, with 06-55-04 microcode revision 0x2000064? Thank you.

[hmh]

easy-rh: oops, sorry about that! wrong names but correct cpuids: tests were done on 0x806e9, 0x906e9, 0x906ea. None show the reboot issue.
(edited the incorrect post, it now has the proper processor names).