XSA-273: L1 Terminal Fault

[DemiMarie]

Qubes OS version:

All

Affected component(s):

Xen, dom0 kernel

---

Steps to reproduce the behavior:

Run cat /sys/devices/system/cpu/vulnerabilities/l1tf in dom0

Expected behavior:

$ cat /sys/devices/system/cpu/vulnerabilities/l1tf
Mitigation: PTE Inversion
VMX: SMT disabled
L1D cache flushes

Actual behavior:

$ cat /sys/devices/system/cpu/vulnerabilities/l1tf
cat: /sys/devices/system/cpu/vulnerabilities/l1tf: No such file or directory

General notes:

This indicates that Qubes is vulnerable to L1 Terminal Fault attacks, which can result in compromise of arbitrary data located in any Qubes domain. Since this includes LUKS keys, this cannot be fully mitigated by not running sensitive and untrusted code simultaneously.

This bug is already known to affect Xen (XSA-273), and has already been brought up on the qubes-users mailing list. As such, it is rather shocking that no QSB has been issued, nor does the XSA tracker make any mention of this XSA.

---

Related issues:

[andrewdavidwong]

This bug is already known to affect Xen (XSA-273), and has already been brought up on the qubes-users mailing list. As such, it is rather shocking that no QSB has been issued, nor does the XSA tracker make any mention of this XSA.

You must have missed this:

https://groups.google.com/d/msg/qubes-users/Isn_hko7tQs/PcqIuUleEQAJ

[DemiMarie]

Yes, I did miss it. That makes sense. How difficult would it be to preemptively push out an updated Xen package?

…

On Mon, Aug 27, 2018, 2:31 AM Andrew David Wong ***@***.***> wrote: This bug is already known to affect Xen (XSA-273), and has already been brought up on the qubes-users mailing list. As such, it is rather shocking that no QSB has been issued, nor does the XSA tracker make any mention of this XSA. You must have missed this: https://groups.google.com/d/msg/qubes-users/Isn_hko7tQs/PcqIuUleEQAJ — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#4252 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGGWB697pMkImntIdBrbcVAaHrlZVG0gks5uU5JWgaJpZM4WNHoa> .

[andrewdavidwong]

Yes, I did miss it. That makes sense. How difficult would it be to
preemptively push out an updated Xen package?

You mean before @marmarek gets back? I don't have any way to do that, so unless someone else like @woju or @omeg does, it won't happen before then.

[HW42]

Now, there's QubesOS/qubes-vmm-xen#45 for people who can/want build it themselves.

You mean before @marmarek gets back? I don't have any way to do that, so unless someone else like @woju or @omeg does, it won't happen before then.

In principle they should both be able to do so since their keys are in https://github.com/QubesOS/qubes-builder/blob/master/qubes-developers-keys.asc, so the builder infrastructure should accept tags signed by them.

[HW42]

@DemiMarie: AFAIU the dom0 kernel doesn't need to be patched.

[lunarthegrey]

Packages got pushed 6 hours ago that set smt=off in your grub configuration. QubesOS/updates-status#642

Does anyone know the impact this will have on performance? I just updated to the new xen binaries from the qubes-dom0-current-testing repository.

Edit: found this https://access.redhat.com/security/vulnerabilities/L1TF-perf

[andrewdavidwong]

We have just published Qubes Security Bulletin (QSB) #43: L1 Terminal Fault speculative side channel (XSA-273).

[ghost]

FWIW performance is catastrophic after the l1tf updates on my thinkpad T450s (i7-5600). What used to be an expensive "state-of-the-art" laptop 2 years ago is now as useful as an electrical heater, to the point that I'm considering ditching all those security fixes and live with the vulnerabilities :(

Eg.:

• without apps or cpu-intensive background services running in VMs, power consumption doesn't drop below 11W, compared to ~ 6W before (which is still quite higher than the power consumption of stock fedora, but that's another topic).
• loadavg is > 0.30
• xentop doesn't show anything striking but the fan is always on, between mid to full speed, which never used to be the case.

I'll try to disable the "fixes" one by one when time allows to see what causes the high loadavg, which is probably the reason for the increased power consumption and the fan kicking in. (Meanwhile, any suggestions welcome).

[marmarek]

R3.2 or R4.0? In case of R3.2, make sure VMs use updated kernel, otherwise fallback mitigation may kick in, which greatly reduce performance (look for L1TF-vulnerable L1e ,,, - Shadowing in xl dmesg).

[ghost]

Sorry, I'm on R4.0 (totally forgot there was still R3.2, I'm using R4.0 since rc2).

Dom0 and templatesVMs are updated with the -testing repos.

After a reboot (as I'm typing this comment), I have sys-{net,firewall,usb} VMs + 2 fedora-28 AppVMs running (one appvm with firefox with only one tab open - github - and one appvm with thunderbird):

• loadavg (dom0): 0.34
• top (dom0): ~99% idle
• xentop shows that sys-usb-dm and sys-net-dm use ~4.5% cpu, dom0 uses ~3% and the rest is negligible, at 0.1-0.2% for each line.
• power consumption on battery (with everything set to min power - xen, sata link, usb, ..): 11.5W, fan running at low-middle speed.
• no additional peripherals are plugged (usb, ...).

[lunarthegrey]

I have the same CPU as @taradiddles and I've noticed an increase in heat coming from the laptop as well (I'm running 4.0). I was expecting this though, as disabling SMT is going to make the CPU work harder. I wonder if we'll ever be able to enable SMT safely again?

[ghost]

I'm not convinced SMT (or lack thereof) is the culprit here: SMT helps in some cases but may also be detrimental in others (eg. scientific simulations, hpc, ...).
Also, from the redhat doc you linked:

Disabling HT in environments with less than ~50% cpu utilization usually showed no performance impact. Disabling HT in environments with higher cpu utilization yields a serious performance drop

(I clearly should be under 50% cpu utiliziation with only a browser + mail client appvms).

I'll try to turn SMT back on later today for tests.

[lunarthegrey]

@taradiddles Turning off SMT on hypervisors has a larger performance impact I believe. I'm not 100% sure, so don't quote me on that..

[ghost]

@lunarthegrey: I guess you're right...

I just did a quick test with smt=on: the laptop's fan stays off or runs at very low speed, and power consumption is back to ~6W on battery.

But quite oddly, loadavg and xentop "metrics" are worse (loadavg @ 0.80, xentop shows ~20% usage by dom0, and ~14% for sys-{usb,net}-dm, top shows 96% idle). No idea how those were before the updates, but those metrics don't seem to be meaningful/representative of the performance problem.
Measuring the time spent by the system doing context switches may provide a better metric. Which makes me think of trying to lower each VM's nb. of cpus to 1 and pin the physical cores to each running VM. That'll be for the next test... (I'll edit the post accordingly).

[airelemental]

With smt=on, I see similar xentop cpu usage of ~14% for *-dm.
With smt=off, during some sessions, xentop halves all the cpu usage(?).
Overall, I haven't seen any deterioration of my user experience with smt=off. (Maybe because I have a quad-core cpu and hardly ever use >400% cpu before.)

[ghost]

@airelemental : are you using a laptop ? If yes, it may be interesting to compare the power consumption on battery with both smt=on and smt=off since it's the only "reliable" metric I've found for now.

I've kept smt=on since my last comment because I can't work on my laptop with smt=off, but I've tried to disable two theads (one on each core) from the cpu pool, with xl cpupool-cpu-remove Pool-0 XX (XX = 1 and 3 resp.). My reasoning is that using only one thread per core should be the same as disabling HT. However I'm not seeing any performance degradation so I expect to be wrong - otherwise xen's people would have disabled the cpu threads instead of turning off HT completely. It'd be interesting to hear what @marmarek thinks about that...

[airelemental]

On my laptop, my power consumption is either unchanged or maybe somewhat improved compared to before, but nowhere near a factor of 2.

[eug48]

Does the stubdom still need to be updated? Since this update I've started getting intermittent crashes of the "-dm" stubdom of a Window 10 VM.

xl dmesg:
(XEN) d30 L1TF-vulnerable L1e 0000000008800000 - Crashing
(XEN) domain_crash called from /home/user/rpmbuild/BUILD/xen-4.8.4/xen/include/asm/shadow.h:185
(XEN) Domain 30 (vcpu#1) crashed on cpu#6:
(XEN) ----[ Xen-4.8.4 x86_64 debug=n Tainted: M ]----
...

...-dm.log:
Linux version 4.14.57-xen-stubdom (user@build-fedora4) (gcc version 6.4.1 20170727 (Red Hat 6.4.1-1) (GCC)) #1 Mon Jul 23 01:07:34 UTC 2018
...

[marmarek]

Yes, that looks to be the case.

@HW42 that's interesting, as stubdomain don't have swap, any idea?

[eug48]

Thanks, the stubdomain hasn't crashed for 2 days now.

[lunarthegrey]

So from what I understand people are experiencing increased heat and batteries don't last as long with smt=off and after a suspend/resume?

Ever since SMT was turned off for me I have noticed increased heat + low battery life. I updated my BIOS so I can turn SMT off in the BIOS and I'm leaving smt=off as is. Does anyone still experience the increased heat + low battery life with SMT set to off in grub and the BIOS? Or is the issue only solved when it is set to on in grub and turned off in the BIOS?

[olekli]

@lunarthegrey I experience no increased CPU temperature with smt=off and hyperthreading disabled in BIOS. Also other issues like xenpm erroring on some cores do not appear.

Oddly enough, with smt=off and hyperthreading enabled, lscpu shows 1 thread per core. But with hyperthreading disabled, it shows 2.

[esote]

@andrewdavidwong This issue was originally opened to have Qubes address L1TF. Per the description, in dom0:

$ cat /sys/devices/system/cpu/vulnerabilities/l1tf
Mitigation: PTE Inversional

So, in that sense, it is now fixed.

For issues related to the performance hit and temperature increase, in the future these could be put together as a separate issue -- where we could track the fix for that independent of the actual addition of L1TF mitigation in Qubes. Let me know if I'm wrong here.

[andrewdavidwong]

@esote: It looks like @marmarek is still merging commits from @fepitre for this. I expect to see this issue closed from such a commit when they consider it to be done. Closing before that could be problematic if it results in things being forgotten or inadvertently deprioritized.

[esote]

@andrewdavidwong Ah, good point. I didn't notice the timestamp on those commits. Thanks!

[marmarek]

The commits referenced above are porting the work done for R4.0, into R4.1 devel branches. The original issue here is completed.