Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Safe use of Hyperthreading when Xen stable includes new sched-gran parameter #5547

Open
trueriver opened this issue Dec 29, 2019 · 18 comments
Open
Labels
C: Xen P: default security T: enhancement
Milestone

Comments

@trueriver
Copy link

trueriver commented Dec 29, 2019

Qubes OS version:

R 4.0.1
All current versions as of 1 Jan 2020

Affected component(s) or functionality:

Xen, hyperthreading, Intel


Steps to reproduce the behavior:

Run Qubes on machine with hyperthreading (HT) hardware

Expected or desired behavior:

HT should ideally be available

Actual behavior:

HT disabled by default

General notes:

HT has been deliberately disabled for security reasons, and with the current relaease versions of Xen this is sensible. The reason is that there exist several exploits on Intel kit that involve abuse of shared cache, etc, when one thread in a CPU is running on one VM and another on another.

However, the Xen wizards are on the case, and they have a fix that will ensure that all the threads running on a cpu are allocated together. This means that the only software exposed by such exploits will be software that already has access to that machine. This may reduce the attack surface to an acceptable level for at least some users.

The patch is now included in some unstable branches of Xen, and is invoked by the command line parameters

smt on sched-gran=core

(or socket or cpu). My request is that this is implemented by Qubes, but ONLY once we start using a Xen version that includes this feature.

However, it is currently (Jan 2020) too soon to change the current behaviour, as the versions of Xen currently in use ignore this combination without warning.

My request therefore is that this is assigned a sensibly long timescale.


I have consulted the following relevant documentation:

https://www.slideshare.net/xen_com_mgr/xpdds19-core-scheduling-in-xen-jrgen-gro-suse

https://patchwork.kernel.org/cover/11086677/

https://xenbits.xen.org/docs/unstable/misc/xen-command-line.html#sched-gran-x86

(NOTE the above path is in the "unstable" branch)

I am aware of the following related, non-duplicate issues:

@trueriver
Copy link
Author

trueriver commented Dec 29, 2019

Typo: the command line params should be

smt=on sched-gran=core

@andrewdavidwong andrewdavidwong added C: Xen P: default security T: enhancement labels Dec 30, 2019
@andrewdavidwong andrewdavidwong added this to the Far in the future milestone Dec 30, 2019
@marmarek
Copy link
Member

marmarek commented Jan 5, 2020

Generally I agree, but only when it gets stable enough. And probably with some option to enable/disable it. While Xen 4.13 do include this feature it is marked as "experimental", which among other things, means it does not receive security support.
Additionally, while it should mitigate cross-VM attacks using HT-related hardware bugs, it will expose in-VM software. It isn't strictly within Qubes security model (we consider in-VM code always untrusted, even when running within some extra sandbox), but preventing in-VM security boundaries from work isn't exactly a good idea.
Some workaround for this issue is to set vcpus=1 for VMs where HT would have unintended consequences. This still allows using HT for other VMs, where cross-process leak isn't really an issue (like, build environment).

@trueriver
Copy link
Author

trueriver commented Jan 5, 2020

@marmarek
Copy link
Member

marmarek commented Jan 5, 2020

is that if Xen was started with smt=yes, then setting vcpus=1 could mean that you get only "half a core", with the other half being run being available for some malicious exploit launched by another VM.

With thread scheduling, that would be true (and this is why we set smt=off), but with core scheduling, the other thread would be idle - effectively not taking advantage of HT for some VMs.

@trueriver
Copy link
Author

trueriver commented Jan 5, 2020

@trueriver
Copy link
Author

trueriver commented Jan 5, 2020

@dylangerdaly
Copy link

dylangerdaly commented Dec 22, 2020

Has SMT been compiled out of Xen 4.14 (Qubes 4.1)?

When I switch it on the kernel fails to load and I just get a black screen

@v6ak
Copy link

v6ak commented Feb 22, 2021

As long as Xen can propagate the CPU topology, the VM could to handle the SMT securely even with multiple cores. Essentially, I see three ways that guest could use:

a. No SMT. (For paranoid?)
b. In-process SMT. (Fair isolation)
c. Full SMT. (No in-VM isolation, maximum performance)

Actually, the slide 14 on https://www.slideshare.net/xen_com_mgr/xpdds19-core-scheduling-in-xen-jrgen-gro-suse suggests suggests that this has been considered:

The guest might want to mitigate against cpu bugs via core-aware scheduling

@ejose19
Copy link

ejose19 commented May 12, 2021

The guest might want to mitigate against cpu bugs via core-aware scheduling

As long as Xen can propagate the CPU topology, the VM could to handle the SMT securely even with multiple cores. Essentially, I see three ways that guest could use:

a. No SMT. (For paranoid?)
b. In-process SMT. (Fair isolation)
c. Full SMT. (No in-VM isolation, maximum performance)

Actually, the slide 14 on https://www.slideshare.net/xen_com_mgr/xpdds19-core-scheduling-in-xen-jrgen-gro-suse suggests suggests that this has been considered:

The guest might want to mitigate against cpu bugs via core-aware scheduling

Seems like core scheduling work is being done as well on kernel side, which as you mentioned, if Xen can propagate the CPU topology it would provide a safer in-VM isolation.

@faerbersteve
Copy link

faerbersteve commented Aug 4, 2021

It seems to work after removing the SMT=off from the command parameter.
In xenpm I see 8 cores with all details.
Before it was showing details only for every second core.

In my case it's Hyperthreading (Intel i7-4720HQ)
Xen 4.14

@marmarek
Copy link
Member

marmarek commented Aug 4, 2021

It seems to work after removing the SMT=off from the command parameter.

Nope. That only re-enabled hyper-threading, but did not enabled core scheduling. This is the configuration where many speculative bugs will be able to steal the data from other VMs! What you need, is to additionally add sched-gran=core parameter.

@faerbersteve
Copy link

faerbersteve commented Aug 5, 2021

thanks for this important hint
With this additional parameter QubesOS is still running
and now with Hyperthreading🥳

Hope this setting will soon be part of Qubes.

@turkja
Copy link

turkja commented Sep 1, 2021

What you need, is to additionally add sched-gran=core parameter.

Does it work with 4.14 that ships now with Qubes 4.1? What about 4.0?

IMO, disabling HT should be optional, advanced security feature to those who need it. On one of my laptops, the performance was reduced significantly with HT disabled. I didn't understand first what was going on until I saw the core count. On another laptop, I don't see big difference. Maybe this depends on the CPU and/or Xen version?

Problematic laptop is running Tiger Lake CPU with 4/8 cores on Qubes 4.1 w/ default Xen 4.14. Once I enabled HT, this machine got new life. With HT disabled, there was some strange lockups.

@pefu
Copy link

pefu commented Feb 10, 2022

I own a laptop from 2013 with a Intel(R) Core(TM) i7-3540M CPU. Adding the parameter sched-gran=core to the kernel CMDLINE did not enable HyperThreading in Qubes-OS 4.0.4 . HT used to work out of the box in Ubuntu 14.04 which was installed on this machine before I switched to Qubes-OS two years ago.
I will test the hyper threading behaviour of Qubes-OS 4.1 on this machine soon.

@drewlander
Copy link

drewlander commented Feb 15, 2022

I added smt=on sched-gran=core and its working just fine on 4.1

@PetrVladimirov
Copy link

PetrVladimirov commented Jul 11, 2022

Cannot switch on SMT on Qubes 4.1 (5.10.112-1.fc32.qubes.x86_64):

  1. Added smt=on with and without sched-gran=core in /etc/default/grub:
    GRUB_CMDLINE_XEN_DEFAULT="console=none smt=on dom0_mem=min:1024M dom0_mem=max:4096M ucode=scan gnttab_max_frames=2048 gnttab_max_maptrack_frames=4096"
  2. Updated grub config sudo grub2-mkconfig -o /boot/grub2/grub.cfg
  3. Rebooted
  4. xl info still shows:
...
threads_per_core       : 1
...
xen_commandline        : placeholder console=none dom0_mem=min:1024M dom0_mem=max:4096M ucode=scan smt=off gnttab_max_frames=2048 gnttab_max_maptrack_frames=4096 no-real-mode edd=off
...

Can please someone suggest what I need to do to finally get it applied?

@tzwcfq
Copy link

tzwcfq commented Jul 12, 2022

There may be multiple GRUB_CMDLINE_XEN_DEFAULT lines in /etc/default/grub and there may be smt=off there but I don't know which one will have priority in this case.
Or maybe you're using EFI and you need to update grub with this command instead:
sudo grub2-mkconfig -o /boot/efi/EFI/qubes/grub.cfg

@PetrVladimirov
Copy link

PetrVladimirov commented Jul 13, 2022

There may be multiple GRUB_CMDLINE_XEN_DEFAULT lines in /etc/default/grub and there may be smt=off there but I don't know which one will have priority in this case. Or maybe you're using EFI and you need to update grub with this command instead: sudo grub2-mkconfig -o /boot/efi/EFI/qubes/grub.cfg

That was the reason! Thank you very much for your advice!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C: Xen P: default security T: enhancement
Projects
None yet
Development

No branches or pull requests