Power Consumption 2-3x after first suspend/resume

[AndrewX192]

Qubes OS version
R4.0.2-rc1

Affected component(s) or functionality
Power Management (Xen)
Brief summary
Power usage 2-3X after first suspend/resume cycle

To Reproduce
(Tested on Lenovo T470 and T490)

1. Turn off laptop
2. Turn on laptop and login to desktop
3. Note power usage using powertop, and look at power consumption - you can wait as long as you like for sampling
4. Suspend computer by closing lid or through system menu
5. Resume computer
6. Note that power usage reported in powertop is 2-3x what it was before, and is sustained until the machine is rebooted

Expected behavior
Power management is restored after resumption, and not materially different than before putting laptop to sleep.

Actual behavior
Laptop consumes 2-3x power consumption in watts before suspend

Screenshots
N/A

Additional context
Not sure how many systems this impacts. I lived with this for years on the T470, but when I got the T490 the symptoms were way more noticeable and I can't carry around a spare battery anymore to mitigate it.

Solutions you've tried

• Setting xempm cpufreq to "ondemand"
• Using tlp in dom0 before/after suspend
• Testing on more than one device (T470/T490)
• Updating to kernel-latest

Relevant documentation you've consulted

• https://wiki.xenproject.org/wiki/Xenpm_command
• https://www.qubes-os.org/doc/newer-hardware-troubleshooting/
• https://groups.google.com/forum/#!searchin/qubes-users/T470%7Csort:date/qubes-users/eqhw02hHtCk/zf77k7q5EwAJ
• https://groups.google.com/forum/#!searchin/qubes-users/T490%7Csort:date/qubes-users/Z0Kfm53zMxQ/oRRR155qDQAJ

Related, non-duplicate issues
none

[marmarek]

Try disabling hyperthreading in BIOS. It may be related to a software method used to disable sibling threads and how it interact with suspend/resume. I've seen related patches in newer Xen version - if that helps, we it may be worth considering a backport.

[AndrewX192]

So, I thought I had disabled Hyperthreading, but it appears that it was still enabled. I turned it on and back off, and now it appears to be disabled. Battery life is significantly better.

[alpha-and-omega]

@marmarek Disabling Hyperthreading solved my #5267 problem too, thanks! Please backport!

[marmarek]

Unfortunately there quite a bit differences between Xen 4.8 and 4.13 and the backport is complex (xen-project/xen@eb91201107, xen-project/xen@8797d20). Given there is an easy workaround (disabling hyperthreading in BIOS, instead of in software), it's safer to go this way. Note we do disable hyperthreading anyway (by not using secondary threads), so you don't pay any performance cost. And actually the code to "disable it" with software has the issues after resume, so by disabling it in BIOS you avoid the problem.

[m-v-b]

Hello @marmarek! My laptop's BIOS does not offer an option to disable hyperthreading, so I cherry-picked/back-ported 5 commits to xen-4.8, which resolved the higher-power-consumption-after-suspend-to-RAM issue according to powertop's power measurements. (For the record, powertop appears to work fine in an HVM, but not in PV mode.)

Would you be interested in a pull request to qubes-vmm-xen to resolve this issue in software in Qubes OS 4.0 for people whose laptops do not have a hyperthread on/off switch?

Here is a list of the commits I cherry-picked:

1. 8d88eacb3b390984e2c483d75d8f40b3ec4be67c ("xen: add helper for calling notifier_call_chain() to common/cpu.c")
2. 51c79e943fb3f9a746181f8b8415cf2baa5d26bd ("xen: add new cpu notifier action CPU_RESUME_FAILED")
3. 6870ea9d1fad6fbe27cf3ce5fe093be709ad2668 ("xen/cpupool: simplify suspend/resume handling")
4. eb912011076e76f6c5e4013616a61ba670e7fc15 ("x86/ACPI: re-park previously parked CPUs upon resume from S3")
5. b0000b128adb07f4107c6e324d32ab025a73a6c8 ("sched: populate cpupool0 only after all cpus are up")

(It is possible that the 5th commit is not truly necessary.)

[marmarek]

Yes, if you managed to get it compile and work, feel free to open a PR :)

[m-v-b]

Hello again,

First of all sorry for the delay, and sorry about the wall of text that follows:

During my tests after having cherry-picked the aforementioned commits, I encountered an unusual issue: systemd-sleep (i.e., systemd's helper program which actually requests from the kernel to suspend the system) would occasionally segfault during the resume from suspend to ram. When inspecting the core dump, the disassemble command in gdb would point to an instruction loading from the %fs register, possibly one that loads data from thread-local storage:

dom0 kernel: systemd-sleep[7102]: segfault at 10 ip 00007fb79ac283f9 sp 00007ffd7c1dd508 error 4 in libc-2.24.so[7fb79abb1000+1bd000]

Core was generated by `/usr/lib/systemd/systemd-sleep suspend'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007fb79ac283f9 in ferror () from /lib64/libc.so.6
Missing separate debuginfos, use: dnf debuginfo-install systemd-udev-231-19.fc25.x86_64
(gdb) bt
#0  0x00007fb79ac283f9 in ferror () from /lib64/libc.so.6
#1  0x00007fb79b4d1f3c in fflush_and_check () from /usr/lib/systemd/libsystemd-shared-231.so
#2  0x000056456b3d63c8 in main ()
(gdb) disassemble 
Dump of assembler code for function ferror:
   0x00007fb79ac283e0 <+0>:     mov    (%rdi),%edx
   0x00007fb79ac283e2 <+2>:     mov    %rdi,%r9
   0x00007fb79ac283e5 <+5>:     mov    %edx,%eax
   0x00007fb79ac283e7 <+7>:     and    $0x8000,%eax
   0x00007fb79ac283ec <+12>:    jne    0x7fb79ac284a0 <ferror+192>
   0x00007fb79ac283f2 <+18>:    mov    0x88(%rdi),%r8
=> 0x00007fb79ac283f9 <+25>:    mov    %fs:0x10,%r10
   0x00007fb79ac28402 <+34>:    cmp    0x8(%r8),%r10

Upon debugging this further, I noticed that this occurs because the %fs register (i.e., the FS segment index register) is clobbered by Xen to the value 0x9b00. Having a non-zero value in %fs appears to cause the kernel to re-load the FS segment when context switching to systemd-sleep running in dom0 userspace, which eventually causes a segfault:

Jul 08 12:54:38 kernel: Enabling non-boot CPUs ...
Jul 08 12:54:38 kernel: installing Xen timer for CPU 1
Jul 08 12:54:38 kernel: xxx: save_fsgs: cpu:0 task:systemd-sleep saved fsindex != 0 (0x9b00)
Jul 08 12:54:38 kernel: xxx: load_seg_legacy: cpu:0 prev_task:systemd-sleep next_task:swapper/0 next_index: 0 <= 3
Jul 08 12:54:38 kernel: xxx: load_seg_legacy: cpu:0 prev_task:systemd-sleep next_task:swapper/0 next_base == 0. next_index:0x0 CPU has X86_BUG_NULL_SEG
Jul 08 12:54:38 kernel: xxx: load_seg_legacy: loading segments. cpu:0. prev_task:swapper/0: next_task:systemd-sleep May cause a segfault... next_index: 39680 > 3 (0x9b00)
...
Jul 08 12:54:38 kernel: PM: suspend exit
Jul 08 12:54:38 kernel: systemd-sleep[7102]: segfault at 10 ip 00007fb79ac283f9 sp 00007ffd7c1dd508 error 4 in libc-2.24.so[7fb79abb1000+1bd000]
Jul 08 12:54:38 kernel: Code: 41 89 48 04 eb e1 66 2e 0f 1f 84 00 00 00 00 00 90 8b 17 49 89 f9 89 d0 25 00 80 00 00 0f 85 ae 00 00 00 4c 8b 87 88 00 00 00 <64> 4c 8b 14 25 10 00 00 00 4d 3b 50 08 0f 84 a4 00 00 00 be 01 00

The fix is thankfully very simple: We can simply stash the original value of the %fs register prior to suspend, and restore it after the resume is complete.

Here is the section of code in Xen that clobbers the %fs register. Quoting from xen/arch/x86/boot/wakeup.S

43	        # boot trampoline is under 1M, and shift its start into
44	        # %fs to reference symbols in that area
45	        mov     wakesym(trampoline_seg), %fs
46	        lidt    %fs:bootsym(idt_48)
47	        lgdt    %fs:bootsym(gdt_48)

However, I have not been able to explain why the clobbered value of the %fs register intermittently causes issues in dom0 userspace with the cherry-picked patches. After instrumenting the kernel and Xen, I have noticed that the migration/x and swapper/x kernel threads seem to be tainted with the leaked %fs register value as well, even in an unpatched Xen configuration.

All this to say, there appears to be a bug in Xen that interacts negatively with the commits I had proposed to cherry-pick to Qubes OS's version of Xen. I also noticed that recent versions of Xen no longer use the %fs register in the aforementioned part of the same file, so it is possible that recent versions of Xen are not affected by this bug.

@marmarek, would you be open to merging an additional patch that stashes and restores the %fs register as part of a pull request?