Update microcode_ctl package for R3.2

[phagara]

Qubes OS version:

R3.2

Affected component(s):

Intel CPU microcode package for dom0 (microcode_ctl)

---

Steps to reproduce the behavior:

Expected behavior:

Actual behavior:

General notes:

Since the Fedora version used in R3.2 dom0 is long since EOL, upstream will not update the microcode_ctl package. Intel recently released a new batch of microcode updates after pulling the previous one due to some bugs. These new microcode files should contain workarounds for some variant(s) of the Spectre/Meltdown vulnerabilities, so it would be nice to have them in R3.2 too (especially since a lot of older hardware will not be getting BIOS updates containing the new microcode).

This would require downloading the new microcode files from Intel and building a new version of microcode_ctl package with them. Perhaps the builds for newer Fedora releases could be useful too.

---

Related issues:

[HW42]

For R4.0 (R3.2 WIP):

• QubesOS/qubes-vmm-xen#33
• https://github.com/HW42/qubes-intel-microcode

Attention: The microcode_ctl packages contains non-free code. But we already distribute it through the Fedora package in the installer.

To actually use it we need to pass ucode=scan to Xen. @marmarek: How should we do this? AFAICS currently the grub/efi parameters are set once by the installer ...

[HW42]

For R4.0 (R3.2 WIP):

• QubesOS/qubes-vmm-xen#33
• https://github.com/HW42/qubes-intel-microcode

Attention: The microcode_ctl packages contains non-free code. But we already distribute it through the Fedora package in the installer.

To actually use it we need to pass ucode=scan to Xen. @marmarek: How should we do this? AFAICS currently the grub/efi parameters are set once by the installer ...

[3hhh]

There's 2 ways to load Microcode. [1]
The Intel description [3] apparently refers to the legacy one. I'm not sure whether Xen supports the legacy one - it's at least not mentioned in [4].

I think both require certain kernel options to be set in order to work. These are apparently set by Qubes [2].

What is strange however:
Neither /sys/devices/system/cpu/microcode/reload nor /dev/cpu/microcode exist in dom0 on my system and sudo journalctl -b0 | grep micro doesn't mention anything with microcode_ctl installed, i.e. we might have a bug here.
EDIT: Ah ok that's probably QubesOS/qubes-vmm-xen#33

Fyi: cat /proc/cpuinfo | grep microcode tells you the version you're on.

[1] https://github.com/torvalds/linux/blob/master/Documentation/x86/microcode.txt
[2] https://github.com/QubesOS/qubes-linux-kernel/blob/stable-3.18/config
[3] https://downloadcenter.intel.com/download/27591/Linux-Processor-Microcode-Data-File
[4] https://wiki.xenproject.org/wiki/XenParavirtOps/microcode_update

[3hhh]

There's 2 ways to load Microcode. [1]
The Intel description [3] apparently refers to the legacy one. I'm not sure whether Xen supports the legacy one - it's at least not mentioned in [4].

I think both require certain kernel options to be set in order to work. These are apparently set by Qubes [2].

What is strange however:
Neither /sys/devices/system/cpu/microcode/reload nor /dev/cpu/microcode exist in dom0 on my system and sudo journalctl -b0 | grep micro doesn't mention anything with microcode_ctl installed, i.e. we might have a bug here.
EDIT: Ah ok that's probably QubesOS/qubes-vmm-xen#33

Fyi: cat /proc/cpuinfo | grep microcode tells you the version you're on.

[1] https://github.com/torvalds/linux/blob/master/Documentation/x86/microcode.txt
[2] https://github.com/QubesOS/qubes-linux-kernel/blob/stable-3.18/config
[3] https://downloadcenter.intel.com/download/27591/Linux-Processor-Microcode-Data-File
[4] https://wiki.xenproject.org/wiki/XenParavirtOps/microcode_update

[marmarek]

To actually use it we need to pass ucode=scan to Xen. @marmarek: How should we do this? AFAICS currently the grub/efi parameters are set once by the installer ...

For grub, we can do similar thing as with automatic USBVM setup: append to /etc/default/grub:

GRUB_CMDLINE_XEN_DEFAULT="$GRUB_CMDLINE_XEN_DEFAULT ucode=scan"

For EFI, we use sed on /boot/efi/EFI/qubes/xen.cfg...

Probably should be part of %postinst on xen-hypervisor. And make sure to not redo it when the setting is already there.

[marmarek]

To actually use it we need to pass ucode=scan to Xen. @marmarek: How should we do this? AFAICS currently the grub/efi parameters are set once by the installer ...

For grub, we can do similar thing as with automatic USBVM setup: append to /etc/default/grub:

GRUB_CMDLINE_XEN_DEFAULT="$GRUB_CMDLINE_XEN_DEFAULT ucode=scan"

For EFI, we use sed on /boot/efi/EFI/qubes/xen.cfg...

Probably should be part of %postinst on xen-hypervisor. And make sure to not redo it when the setting is already there.

[marmarek]

Probably should be part of %postinst on xen-hypervisor. And make sure to not redo it when the setting is already there.

or microcode_ctl package.
According to xen docs, ucode=scan will not work with EFI, that one needs ucode=<filename> in /boot/efi/EFI/qubes/xen.cfg.

[marmarek]

Probably should be part of %postinst on xen-hypervisor. And make sure to not redo it when the setting is already there.

or microcode_ctl package.
According to xen docs, ucode=scan will not work with EFI, that one needs ucode=<filename> in /boot/efi/EFI/qubes/xen.cfg.

[marmarek]

But despite documentation, ucode=scan command line option works with EFI too.

[marmarek]

But despite documentation, ucode=scan command line option works with EFI too.

[marmarek]

The current microcode_ctl package breaks installation: https://openqa.qubes-os.org/tests/392#step/install_do_user/8
Testing a fix here: https://github.com/marmarek/qubes-intel-microcode/tree/postinstall-fix

[marmarek]

The current microcode_ctl package breaks installation: https://openqa.qubes-os.org/tests/392#step/install_do_user/8
Testing a fix here: https://github.com/marmarek/qubes-intel-microcode/tree/postinstall-fix

[marmarek]

Above fix works. Also adding ucode=scan done.

[marmarek]

Above fix works. Also adding ucode=scan done.

[HW42]

We can avoid using the rather hacky editing of the command line by just changing the default in Xen (you can still disable microcode loading with ucode=0). Didn't push yet because something with the microcode loading is broken with Xen 4.6 (system freeze or crash).

[HW42]

We can avoid using the rather hacky editing of the command line by just changing the default in Xen (you can still disable microcode loading with ucode=0). Didn't push yet because something with the microcode loading is broken with Xen 4.6 (system freeze or crash).

[marmarek]

I'd rather limit number of our patches to Xen, rather than adding new ones. Unless that could be accepted upstream.

[marmarek]

I'd rather limit number of our patches to Xen, rather than adding new ones. Unless that could be accepted upstream.

[3hhh]

This looks pretty Intel-specific so far.

Was support for AMD ucode updates considered?

[3hhh]

This looks pretty Intel-specific so far.

Was support for AMD ucode updates considered?

[phagara]

@3hhh microcode_ctl package is "obsolete", it's being used only because Intel does not (yet?) provide microcode blobs compatible with Linux kernel ucode loader (AMD apparently does). In Qubes and other RPM-based distros, AMD ucode is provided by the linux-firmware package. Still, Qubes will need to copy the relevant ucode files to /boot and tell Xen about it, exactly like discussed in the comments above.

[phagara]

@3hhh microcode_ctl package is "obsolete", it's being used only because Intel does not (yet?) provide microcode blobs compatible with Linux kernel ucode loader (AMD apparently does). In Qubes and other RPM-based distros, AMD ucode is provided by the linux-firmware package. Still, Qubes will need to copy the relevant ucode files to /boot and tell Xen about it, exactly like discussed in the comments above.

[mirrorway]

The new microcode breaks suspend on my corebooted Thinkpad (4.0rc5). Instead of waking up from sleep, it restarts.

[mirrorway]

The new microcode breaks suspend on my corebooted Thinkpad (4.0rc5). Instead of waking up from sleep, it restarts.

[marmarek]

@mirrorway I guess its rather xen-4.8.3-4, not microcode itself. See #3738

[marmarek]

@mirrorway I guess its rather xen-4.8.3-4, not microcode itself. See #3738

[h01ger]

On Wed, Mar 28, 2018 at 07:20:09AM -0700, mirrorway wrote: The new microcode causes my corebooted Thinkpad on 4.0rc5 to restart instead of waking up from sleep.

it would be great if you and the original submitter could state which thinkpad models you are using and also which coreboot version. Thanks!

…

-- cheers, Holger

[h01ger]

On Wed, Mar 28, 2018 at 07:20:09AM -0700, mirrorway wrote: The new microcode causes my corebooted Thinkpad on 4.0rc5 to restart instead of waking up from sleep.

it would be great if you and the original submitter could state which thinkpad models you are using and also which coreboot version. Thanks!

…

-- cheers, Holger

[mirrorway]

T530, coreboot+grub from Dec 2017 - CBET4000 4.6-2454-g33cd28e7ac

It seemed like it was caused by the microcode update, which upgraded my ucode to 0x1f.
xen-4.8.3-4 resumes fine after I remove ucode=scan (reverting to ucode 0x1b).

Typo: it's grub, not seabios.

[mirrorway]

T530, coreboot+grub from Dec 2017 - CBET4000 4.6-2454-g33cd28e7ac

It seemed like it was caused by the microcode update, which upgraded my ucode to 0x1f.
xen-4.8.3-4 resumes fine after I remove ucode=scan (reverting to ucode 0x1b).

Typo: it's grub, not seabios.

[marmarek]

What about xen-4.8.3-3 while keeping ucode=scan?

[marmarek]

What about xen-4.8.3-3 while keeping ucode=scan?

[mirrorway]

How do I downgrade xen? Installing the downgraded version doesn't work:

$ sudo qubes-dom0-update xen-4.8.3-3.fc25.x86_64
Using sys-whonix as UpdateVM to download updates for Dom0; this may take some time...
qubes-dom0-current-testing                                  | 3.6 kB  00:00     
--> Running transaction check
---> Package xen.x86_64 2001:4.8.3-3.fc25 will be installed
--> Processing Dependency: xen-runtime = 4.8.3-3.fc25 for package: 2001:xen-4.8.3-3.fc25.x86_64
--> Finished Dependency Resolution
xen-4.8.3-3.fc25.x86_64.rpm                                 |  77 kB  00:00     
find: '/var/lib/qubes/dom0-updates/var/cache/yum': No such file or directory
Qubes OS Repository for Dom0                     24 MB/s |  26 kB     00:00    
No package xen-4.8.3-3.fc25.x86_64 available.
Error: Unable to find a match.

[mirrorway]

How do I downgrade xen? Installing the downgraded version doesn't work:

$ sudo qubes-dom0-update xen-4.8.3-3.fc25.x86_64
Using sys-whonix as UpdateVM to download updates for Dom0; this may take some time...
qubes-dom0-current-testing                                  | 3.6 kB  00:00     
--> Running transaction check
---> Package xen.x86_64 2001:4.8.3-3.fc25 will be installed
--> Processing Dependency: xen-runtime = 4.8.3-3.fc25 for package: 2001:xen-4.8.3-3.fc25.x86_64
--> Finished Dependency Resolution
xen-4.8.3-3.fc25.x86_64.rpm                                 |  77 kB  00:00     
find: '/var/lib/qubes/dom0-updates/var/cache/yum': No such file or directory
Qubes OS Repository for Dom0                     24 MB/s |  26 kB     00:00    
No package xen-4.8.3-3.fc25.x86_64 available.
Error: Unable to find a match.

[phagara]

Normally, sudo qubes-dom0-update --action=downgrade xen xen-hvm xen-libs xen-runtime (R4 might have different dependencies) would do the trick (you may also specify the full version for each package if just the previous update won't cut it). However, this might not work for you using whonix since I'm getting the following error using a debian-9 based VM:

ERROR: yum version installed in VM updatevm does not suppport --downloadonly option
ERROR: only 'install' and 'upgrade' actions supported (downgrade not)

As a workaround for when you get the above error, cd into /var/lib/qubes/updates/rpm or similar and install manually using yum downgrade xen*.rpm in dom0.

[phagara]

Normally, sudo qubes-dom0-update --action=downgrade xen xen-hvm xen-libs xen-runtime (R4 might have different dependencies) would do the trick (you may also specify the full version for each package if just the previous update won't cut it). However, this might not work for you using whonix since I'm getting the following error using a debian-9 based VM:

ERROR: yum version installed in VM updatevm does not suppport --downloadonly option
ERROR: only 'install' and 'upgrade' actions supported (downgrade not)

As a workaround for when you get the above error, cd into /var/lib/qubes/updates/rpm or similar and install manually using yum downgrade xen*.rpm in dom0.

[mirrorway]

$ ls
xen-4.8.3-3.fc25.x86_64.rpm      xen-libs-4.8.3-3.fc25.x86_64.rpm
xen-hvm-4.8.3-3.fc25.x86_64.rpm  xen-runtime-4.8.3-3.fc25.x86_64.rpm

$ sudo dnf downgrade xen*.rpm
Qubes OS Repository for Dom0                     29 MB/s |  30 kB     00:00    
Error: package python3-xen-2001:4.8.3-4.fc25.x86_64 requires xen-libs = 4.8.3-4.fc25, but none of the providers can be installed.
package xen-runtime-2001:4.8.3-3.fc25.x86_64 requires xen-libs = 4.8.3-3.fc25, but none of the providers can be installed.
package xen-hvm-2001:4.8.3-3.fc25.x86_64 requires xen-libs = 4.8.3-3.fc25, but none of the providers can be installed.
package xen-2001:4.8.3-3.fc25.x86_64 requires xen-runtime = 4.8.3-3.fc25, but none of the providers can be installed
(try to add '--allowerasing' to command line to replace conflicting packages)

$ sudo dnf downgrade --allowerasing xen*.rpm
Qubes OS Repository for Dom0                     29 MB/s |  30 kB     00:00    
Dependencies resolved.
Error: The operation would result in removing the following protected packages: qubes-core-dom0.

Seems like it could be dangerous.

[mirrorway]

$ ls
xen-4.8.3-3.fc25.x86_64.rpm      xen-libs-4.8.3-3.fc25.x86_64.rpm
xen-hvm-4.8.3-3.fc25.x86_64.rpm  xen-runtime-4.8.3-3.fc25.x86_64.rpm

$ sudo dnf downgrade xen*.rpm
Qubes OS Repository for Dom0                     29 MB/s |  30 kB     00:00    
Error: package python3-xen-2001:4.8.3-4.fc25.x86_64 requires xen-libs = 4.8.3-4.fc25, but none of the providers can be installed.
package xen-runtime-2001:4.8.3-3.fc25.x86_64 requires xen-libs = 4.8.3-3.fc25, but none of the providers can be installed.
package xen-hvm-2001:4.8.3-3.fc25.x86_64 requires xen-libs = 4.8.3-3.fc25, but none of the providers can be installed.
package xen-2001:4.8.3-3.fc25.x86_64 requires xen-runtime = 4.8.3-3.fc25, but none of the providers can be installed
(try to add '--allowerasing' to command line to replace conflicting packages)

$ sudo dnf downgrade --allowerasing xen*.rpm
Qubes OS Repository for Dom0                     29 MB/s |  30 kB     00:00    
Dependencies resolved.
Error: The operation would result in removing the following protected packages: qubes-core-dom0.

Seems like it could be dangerous.

[marmarek]

Actually the most relevant package here is xen-hypervisor. Try downgrading just this one.

[marmarek]

Actually the most relevant package here is xen-hypervisor. Try downgrading just this one.

[mirrorway]

No resume issues with xen-hypervisor-4.8.3-3.fc25.x86_64 and updated microcode loaded.

$ rpm -qa | grep '^xen'
xen-libs-4.8.3-4.fc25.x86_64
xen-hvm-4.8.3-4.fc25.x86_64
xen-hypervisor-4.8.3-3.fc25.x86_64
xen-hvm-stubdom-linux-1.0.7-1.fc25.x86_64
xen-runtime-4.8.3-4.fc25.x86_64
xen-4.8.3-4.fc25.x86_64
xen-licenses-4.8.3-4.fc25.x86_64

$ rpm -q microcode_ctl
microcode_ctl-2.1-22.qubes2.fc25.x86_64

$ cat /proc/cpuinfo | grep microcode
microcode	: 0x1f
microcode	: 0x1f
microcode	: 0x1f
microcode	: 0x1f
microcode	: 0x1f
microcode	: 0x1f
microcode	: 0x1f
microcode	: 0x1f

[mirrorway]

No resume issues with xen-hypervisor-4.8.3-3.fc25.x86_64 and updated microcode loaded.

$ rpm -qa | grep '^xen'
xen-libs-4.8.3-4.fc25.x86_64
xen-hvm-4.8.3-4.fc25.x86_64
xen-hypervisor-4.8.3-3.fc25.x86_64
xen-hvm-stubdom-linux-1.0.7-1.fc25.x86_64
xen-runtime-4.8.3-4.fc25.x86_64
xen-4.8.3-4.fc25.x86_64
xen-licenses-4.8.3-4.fc25.x86_64

$ rpm -q microcode_ctl
microcode_ctl-2.1-22.qubes2.fc25.x86_64

$ cat /proc/cpuinfo | grep microcode
microcode	: 0x1f
microcode	: 0x1f
microcode	: 0x1f
microcode	: 0x1f
microcode	: 0x1f
microcode	: 0x1f
microcode	: 0x1f
microcode	: 0x1f

[HW42]

For R3.2: QubesOS/qubes-vmm-xen#36

[HW42]

For R3.2: QubesOS/qubes-vmm-xen#36

[HW42]

For AMD microcode: https://github.com/HW42/qubes-linux-firmware. This also fixes #3796. Of course this contains a lot of non-free code, but like the Intel microcode we already ship an older version of it as part of the installer so this should not be a (new) problem.

@marmarek: I don't have a recent AMD CPU for testing. So unless you happen to be able to test it please upload the linux-firmware package first to unstable so that somebody else can test.

[HW42]

For AMD microcode: https://github.com/HW42/qubes-linux-firmware. This also fixes #3796. Of course this contains a lot of non-free code, but like the Intel microcode we already ship an older version of it as part of the installer so this should not be a (new) problem.

@marmarek: I don't have a recent AMD CPU for testing. So unless you happen to be able to test it please upload the linux-firmware package first to unstable so that somebody else can test.

[HW42]

@marmarek: And you need to upload the files contained in linux-firmware-20180402-83.git8c1e439c.fc26.src.rpm to https://ftp.qubes-os.org/distfiles/.

[HW42]

@marmarek: And you need to upload the files contained in linux-firmware-20180402-83.git8c1e439c.fc26.src.rpm to https://ftp.qubes-os.org/distfiles/.

[marmarek]

linux-firmware package(s) uploaded to unstable repo for R4.0. Anyone with AMD CPU here? If needed, I can upload it also for R3.2.
How to test:

sudo qubes-dom0-update --enablerepo=qubes-dom0-unstable linux-firmware

Then reboot and see if things still works. It may be good idea to wait for xen-4.8.3-5 with fix for QubesOS/qubes-issues#3738

[marmarek]

linux-firmware package(s) uploaded to unstable repo for R4.0. Anyone with AMD CPU here? If needed, I can upload it also for R3.2.
How to test:

sudo qubes-dom0-update --enablerepo=qubes-dom0-unstable linux-firmware

Then reboot and see if things still works. It may be good idea to wait for xen-4.8.3-5 with fix for QubesOS/qubes-issues#3738

[HW42]

@3hhh: You had asked about AMD. Had you time to test the updated linux-firmware packages (see @marmarek's comment directly above)?

---

For anybody who has a test machine with a recent AMD CPU it would be helpful if you could update the xen packages from the testing repo and the install the linux-firmware package from the unstable repo (see above). Then boot with an added Xen option loglvl=all and look for the BTI mitigation messages. Also please test suspend/resume.

[HW42]

@3hhh: You had asked about AMD. Had you time to test the updated linux-firmware packages (see @marmarek's comment directly above)?

---

For anybody who has a test machine with a recent AMD CPU it would be helpful if you could update the xen packages from the testing repo and the install the linux-firmware package from the unstable repo (see above). Then boot with an added Xen option loglvl=all and look for the BTI mitigation messages. Also please test suspend/resume.

[arsenica]

On a fresh install of 4.0 I upgraded dom0 to current-testing, installed linux-firmware from unstable and I added loglvl=all to the Xen boot options.

The hardware is:
AMD ryzen 5 1600
Asrock AB350 pro4
AMD RX570
With HVM, IOMMU and SLAT working

grep -i bti /var/log/xen/console/hypervisor.log
(XEN) BTI mitigations: thunk LFENCE, Others: IBPB RSB_NATIVE RSB_VMEXIT

It will suspend but it doesn't resume properly; the fans and lights come on but nothing is sent to the display. Though that's the case with or without the microcode from unstable. Everything else seems to be working properly.

If anything else needs testing I'd be happy to do it.

[arsenica]

On a fresh install of 4.0 I upgraded dom0 to current-testing, installed linux-firmware from unstable and I added loglvl=all to the Xen boot options.

The hardware is:
AMD ryzen 5 1600
Asrock AB350 pro4
AMD RX570
With HVM, IOMMU and SLAT working

grep -i bti /var/log/xen/console/hypervisor.log
(XEN) BTI mitigations: thunk LFENCE, Others: IBPB RSB_NATIVE RSB_VMEXIT

It will suspend but it doesn't resume properly; the fans and lights come on but nothing is sent to the display. Though that's the case with or without the microcode from unstable. Everything else seems to be working properly.

If anything else needs testing I'd be happy to do it.

[3hhh]

Sorry, I don't run an AMD CPU. I had only noticed that all of the commits before were Intel centric.

I just asked for help for this thread on the users ML:
https://groups.google.com/forum/#!topic/qubes-users/1XCidVWJ-OI

[3hhh]

Sorry, I don't run an AMD CPU. I had only noticed that all of the commits before were Intel centric.

I just asked for help for this thread on the users ML:
https://groups.google.com/forum/#!topic/qubes-users/1XCidVWJ-OI

[sergiomattaa]

I am using:
AMD AM3 Phenom II 1100T X6 6 cores 6 threads
Asus Sabertooth R2.0 BIOS 2901 990FX DDR3
HVM, IOMMU, SLAT
Qubes R4.0, Xen 4.8.3-5, Kernel 4.14.18-1

Normal boot with Xen Boot with ucode=scan log1v1=all, nothing observed

grep -i bti /var/log/xen/console/hypervisor.log shows nothing

[sergiomattaa]

I am using:
AMD AM3 Phenom II 1100T X6 6 cores 6 threads
Asus Sabertooth R2.0 BIOS 2901 990FX DDR3
HVM, IOMMU, SLAT
Qubes R4.0, Xen 4.8.3-5, Kernel 4.14.18-1

Normal boot with Xen Boot with ucode=scan log1v1=all, nothing observed

grep -i bti /var/log/xen/console/hypervisor.log shows nothing

[3hhh]

Thanks a lot to you two for testing!

@marmarek @HW42 Please advise on how to proceed.

[3hhh]

Thanks a lot to you two for testing!

@marmarek @HW42 Please advise on how to proceed.

[awokd]

FWIW these updates don't hurt a corebooted A10-5750M. Microcode isn't getting updated, but I think that's an upstream issue somewhere with the equivalency table (that's originally why I went the route of patching coreboot instead) or fam15 updates aren't yet included. (XEN) BTI mitigations: Thunk LFENCE, Others: RSB_NATIVE RSB_VMEXIT and dom0 kernel: Spectre V2 : Mitigation: Full AMD retpoline showing in the logs.

[awokd]

FWIW these updates don't hurt a corebooted A10-5750M. Microcode isn't getting updated, but I think that's an upstream issue somewhere with the equivalency table (that's originally why I went the route of patching coreboot instead) or fam15 updates aren't yet included. (XEN) BTI mitigations: Thunk LFENCE, Others: RSB_NATIVE RSB_VMEXIT and dom0 kernel: Spectre V2 : Mitigation: Full AMD retpoline showing in the logs.