systemd:amd64 (252-2 -> 252.1-1) brakes suspend/resume

[axet]

systemd version the issue has been seen with

252.1-1

Used distribution

debian bookworm

Linux kernel version used

6.0.0-3-amd64

CPU architectures issue was seen on

x86_64

Component

systemd

Expected behaviour you didn't see

suspend resume to work

Unexpected behaviour you saw

after suspend resume I unable to move any windows. I had to switch to console (alt+f2) and press ctrl+alt+del to reboot

Steps to reproduce the problem

call 'systemctl suspend' or close notebook lid

Additional program output to the terminal or log subsystem illustrating the issue

all hardware logs

* http://linux-hardware.org/?probe=59aa7d31d8

Manually rolling back to 252-2 makes system to work again, windows moves.

EDIT: updated logs https://linux-hardware.org/?probe=b5eb364ac6

[yuwata]

@axet and @Wallaby111 Could you bisect between v252 and v252.1 ? I cannot find suspicious commits...

[axet]

@yuwata I do not see 252.1 tag, What am I missing?

debian changelog saying it is here:

• https://tracker.debian.org/media/packages/s/systemd/changelog-252.1-1

[yuwata]

You can find v252.1 tag at https://github.com/systemd/systemd-stable

[axet]

'make install' breaks keyboard and mouse on my system (no longer pressed, or moved, even alt+f1 no longer works). running 'apt reinstall systemd' makes it unbootable. I'm doing something wrong here. rebuilding systemd more dangerous then rebuilding kernel

EDIT: after full system fresh install, I can confirm: suspend / resume no longer works with recent packages.
EDIT: It works sometimes. I'm no longer sure if it systemd related or just kernel amdgpu bug.

[solsticedhiver]

I am commenting here because other bugs have been closed as duplicates.

I was using suspend-then-hibernate on my desktop with 251.7 and this was working as expected. aka suspending to RAM, and then if not waken up, hibernating after HibernateDelaySec time.

I am very disapointed about the behavior change that seems to use a % of battery to activate the hibernate action. As this would mean, this would not work anymore on a desktop?

Or this a bug? explaining, why suspend-then-hibernate does not hibernate here, now with 252.1 ?

[yuwata]

@solsticedhiver That's already reported and discussed at #25269. And I am already working on the issue. See #25374.

[axet]

Seems debian bookworm linux-image-6.0.0-4-amd64 works fine. I got successfull suspend / resume cycles 10 times in a row. I'm not sure what was wrong. Maybe it was a kernel issue in my case

[n-peugnet]

This is strange as I had a similar problem on Debian Bookworm with 252-2 already (didn't test with 252.1-1).
Also with the linux-image-6.0.0-3-amd64 from Debian (6.0.5-1 really).

And it seems I am not the only one : https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1023483.

I found loginctl terminate-user from another TTY to work in order to get back to the login screen.

[Wallaby111]

Upgrading to the latest Arch kernel seems to have fixed the issue for me as well. Now on 6.0.9-arch1-1 and can both suspend/resume and suspend-then-hibernate/resume with no issues. It did have an issue getting out of betterlockscreen when its auto-lock on suspend service was enabled, but quickly disabling then enabling it fixed that right away.

[yuwata]

Thank you for the reports. So, let's drop the regression tag.

@n-peugnet Could you also test with newer kernels?

[testudomarginata]

Hi,
Debian bookworm linux-image-6.0.0-4-amd64 still has the same issue for me, system is not entering hibernate after HibernateDelaySec (although after reading through the other issues linked from here, this might be new and expected behaviour?) and then the screen remains black on wake-up.
Since downgrading to systemd-251.6-1 fixes this without changing any other package, would this not be considered a systemd regression?
Please let me know if I can supply any more information.
Regards,
Sönke

[axet]

Ok. Bug not complete went out. Now after suspend / resume. My keyboard has 5-10 seconds delay before typing keys. Alt+Tab works fine. Alt+F1 works and console works fine. So weird. Also I got gnome-terminal error in journalctl:

ноя 18 23:40:19 axet-laptop gnome-terminal-[2106]: Events queue growing too big, will start to drop.
ноя 18 23:40:19 axet-laptop gnome-terminal-[2106]: Process Key Event failed: Timeout was reached.
ноя 18 23:40:19 axet-laptop gnome-terminal-[2106]: Events queue growing too big, will start to drop.
ноя 18 23:40:19 axet-laptop gnome-terminal-[2106]: Events queue growing too big, will start to drop.
ноя 18 23:40:19 axet-laptop gnome-terminal-[2106]: Process Key Event failed: Timeout was reached.
ноя 18 23:40:19 axet-laptop gnome-terminal-[2106]: Events queue growing too big, will start to drop.

ноя 19 10:29:34 axet-laptop gedit[31158]: Unable to connect to ibus: Error receiving data: Connection reset by peer
ноя 19 10:29:34 axet-laptop systemd[1]: system-systemd\x2dcoredump.slice: Consumed 3.764s CPU time.
ноя 19 10:29:34 axet-laptop firefox-esr.desktop[26491]: [Parent 26491, Main Thread] WARNING: Process Key Event failed: The connection is closed.: 'glib warni>
ноя 19 10:29:34 axet-laptop firefox-esr.desktop[26491]: [Parent 26491, Main Thread] WARNING: Process Key Event failed: The connection is closed.: 'glib warni>
ноя 19 10:29:34 axet-laptop firefox-esr.desktop[26491]: [Parent 26491, Main Thread] WARNING: Process Key Event failed: The connection is closed.: 'glib warni>
ноя 19 10:29:34 axet-laptop gnome-terminal-[2106]: Process Key Event failed: The connection is closed.
ноя 19 10:29:34 axet-laptop gnome-terminal-[2106]: Process Key Event failed: The connection is closed.
ноя 19 10:29:34 axet-laptop gnome-terminal-[2106]: Process Key Event failed: The connection is closed.
ноя 19 10:29:34 axet-laptop gnome-terminal-[2106]: Process Key Event failed: The connection is closed.
ноя 19 10:29:34 axet-laptop gnome-terminal-[2106]: Process Key Event failed: The connection is closed.
ноя 19 10:29:34 axet-laptop gnome-terminal-[2106]: Process Key Event failed: The connection is closed.
ноя 19 10:29:34 axet-laptop gnome-terminal-[2106]: Process Key Event failed: The connection is closed.
ноя 19 10:29:34 axet-laptop gnome-terminal-[2106]: Process Key Event failed: The connection is closed.
ноя 19 10:29:34 axet-laptop gnome-terminal-[2106]: Process Key Event failed: The connection is closed.
ноя 19 10:29:34 axet-laptop systemd[1]: Closed Load/Save RF Kill Switch Status /dev/rfkill Watch.
ноя 19 10:29:34 axet-laptop gnome-terminal-[2106]: Process Key Event failed: The connection is closed.
ноя 19 10:29:34 axet-laptop gnome-terminal-[2106]: Process Key Event failed: The connection is closed.
ноя 19 10:29:34 axet-laptop gnome-terminal-[2106]: Process Key Event failed: The connection is closed.
ноя 19 10:29:34 axet-laptop gnome-terminal-[2106]: Process Key Event failed: The connection is closed.
``

[yuwata]

@testudomarginata Your issue is not relevant, and is #25269. See also #25356 (comment).

[axet]

Bug get back! All keys get frozen again and I unable to move windows. To summarize: 1) random 2) kernel in-depended 3) keys can be delayed 4) keys can be ignored.

hw-probe

• https://linux-hardware.org/?probe=359c282b27

journalctl after resume 15:30:

• https://pastebin.com/PK6Pd9ak

[Wallaby111]

After reading through the comments on #25269 and seeing @yuwata say that HibernateDelaySeconds= is now hijacked as an interval for measuring discharge rate, I did a little testing. At least on my system, it seems that when going in to suspend-then-hibernate I don't have the resume issues until after that HibernateDelaySeconds= has passed.

I tested this by making "HibernateDelaySeconds=30" in /etc/systemd/sleep.conf, rebooting my laptop, and then entering suspend-then-hibernate and waking it up immediately, then waking it up after a little over 30 seconds. I never had a resume issue after waking it up immediately, but I always did after waiting the 30 seconds. @axet, this could be what makes it appear random to you; repeated testing multiple times in row will work fine, but if you leave it long enough it will have problems resuming.

If someone else could test this to make sure it's not some issue with my specific setup, that'd be really helpful.

Edit: It looks like @yuwata is already working on making some more clear, separate distinctions for HibernateDelaySeconds and the discharge rate measuring interval, so hopefully that will fix the issue, but I'm not sure.

[fkru]

After reading through the comments on #25269 and seeing @yuwata say that HibernateDelaySeconds= is now hijacked as an interval for measuring discharge rate, I did a little testing. At least on my system, it seems that when going in to suspend-then-hibernate I don't have the resume issues until after that HibernateDelaySeconds= has passed.

I tested this by making "HibernateDelaySeconds=30" in /etc/systemd/sleep.conf, rebooting my laptop, and then entering suspend-then-hibernate and waking it up immediately, then waking it up after a little over 30 seconds. I never had a resume issue after waking it up immediately, but I always did after waiting the 30 seconds. @axet, this could be what makes it appear random to you; repeated testing multiple times in row will work fine, but if you leave it long enough it will have problems resuming.

If someone else could test this to make sure it's not some issue with my specific setup, that'd be really helpful.

Edit: It looks like @yuwata is already working on making some more clear, separate distinctions for HibernateDelaySeconds and the discharge rate measuring interval, so hopefully that will fix the issue, but I'm not sure.

Unfortunately, I cannot confirm your observation, as far as the original bug report is concerned.

[n-peugnet]

@yuwata:

Thank you for the reports. So, let's drop the regression tag.

@n-peugnet Could you also test with newer kernels?

So I tried with systemd 252.1-1.

1. I did not yet reproduce the issue with linux-image-6.0.0-4-amd64 (6.0.8-1). EDIT: I just experienced the second behaviour described.
2. I tried to test again with linux-image-6.0.0-2-amd64 (6.0.5-1) and figured out it happened only when my slock service was enabled.

Here is a setup that almost always triggers it:

Setup demo

$ uname -a
Linux X260-NICOLAS 6.0.0-2-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.0.5-1 (2022-10-28) x86_64 GNU/Linux
$ systemd --version
systemd 252 (252.1-1)
+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT -GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY -P11KIT +QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified
$ systemctl cat slock@.service
# /etc/systemd/system/slock@.service
[Unit]
Description=Lock X session using slock for user %i
Before=sleep.target

[Service]
User=%i
Environment=DISPLAY=:0
ExecStartPre=/usr/bin/xset dpms force suspend
ExecStart=/usr/local/bin/slock

[Install]
WantedBy=sleep.target
$ cat /etc/systemd/sleep.conf.d/nicolas.conf
[Sleep]
HibernateDelaySec=10

Triggering

systemctl suspend-then-hibernate

Then wait for 10+ seconds and power on the computer.

Behaviour

The behaviour varies a little between the different occurences.

Usually the screen is left as if slock were not run, but it is completely unresponsive. I can usually only move the mouse and the keyboard does nothing. I can still change TTY using ctrl+alt+f* binbings.

One time slock was visible when loging back in and I can kind of unlock it by typing my password but then gnome-terminal is completely frozen. I can still launch st via dmenu (didn't find other apps working). in this case I can still run systemctl status but systemctl --user status waits forever. This time as I had only one session open I tried loginctl terminate-session 22 to kill it. It allowed me to go back to lightdm login page but didn't correcly lauch my xsession on startup. I had to loginctl terminate-user once again.

Appendices

1. The slock@.service came from Arch wiki and never failed before.

2. Setting HibernateDelaySec=10 is not necessary but I found the issue to appear more consistently with a smaller value.

3. As I said in suspend-then-hibernate: add option to force early hibernate #25269 (comment) I suspect sleep: freeze and thaw user.slice to avoid wasting power/resuming tasks, without affecting the wakeup handler #24336 to be related to this issue (based on its name).

4. Systemd 251-6 worked nicely with the slock service on both linux-image-6.0.0-2-amd64 (6.0.5-1) and linux-image-6.0.0-4-amd64 (6.0.8-1) (and the versions before too obviously).

I will continue to use systemd 252.1-1 with linux-image-6.0.0-4-amd64 (6.0.8-1) and my slock service to see if it appears again. EDIT: It just happened so I will revert to 251.6.

[fkru]

Was it a cold boot after downgrading?

Yes. I downgraded, restarted and then tested. When I upgrade again no restart is needed to brake resume again. sweat_smile

Let me ask again: When you upgrade to 252.2, then shutdown and boot, does the issue still occur? Thx.

[jojoob]

Yes, it does.

[msizanoen1]

Can you check if applying #25662 fixes the problem? (You can find out how to do so by consulting your distribution's packaging documentation)

[axet]

Did't help my system. Keys keep freezing.

Since here no longer 252.1-1 available in debian repos you have to downgrade it manually. To do so compile 251.4 using following example:

dget https://deb.debian.org/debian/pool/main/s/systemd/systemd_251.4-3.dsc

cd systemd-251.4

patch -p0 << EOF
--- ./src/boot/efi/meson.build.old	2022-11-30 21:28:01.540534695 +0300
+++ ./src/boot/efi/meson.build	2022-11-30 21:30:52.705198001 +0300
@@ -221,9 +221,6 @@
 if get_option('debug') and get_option('mode') == 'developer'
         efi_cflags += ['-ggdb', '-DEFI_DEBUG']
 endif
-if get_option('optimization') != '0'
-        efi_cflags += ['-O' + get_option('optimization')]
-endif
 if get_option('b_ndebug') == 'true' or (
    get_option('b_ndebug') == 'if-release' and get_option('buildtype') in ['plain', 'release'])
         efi_cflags += ['-DNDEBUG']
EOF

dpkg-buildpackage -rfakeroot -b -d -us -uc -ui


sudo dpkg -i systemd_251.4-3_amd64.deb  libsystemd-shared_251.4-3_amd64.deb  libsystemd0_251.4-3_amd64.deb libpam-systemd_251.4-3_amd64.deb  systemd-coredump_251.4-3_amd64.deb systemd-timesyncd_251.4-3_amd64.deb systemd-sysv_251.4-3_amd64.deb

[msizanoen1]

Can you execute the following command from SSH when the keys are frozen, and provide the output here?

find /sys/fs/cgroup -name cgroup.freeze -print0 | xargs -0 grep 1

Make sure to reboot the system after installing the patched packages first.

[msizanoen1]

@axet It turns out one of the patches broke cgroup thawing altogether. I just uploaded the fixed patchset. Can you retry applying, rebooting and testing that PR?

[axet]

It worked. But wait. It happened before. Lets wait for a week and see if it is stable during the night. BTW on my system HibernateDelaySec=16h. Never hits the delay.

[msizanoen1]

@axet

It worked. But wait. It happened before. Lets wait for a week and see if it is stable during the night. BTW on my system HibernateDelaySec=16h. Never hits the delay.

If you run any QEMU/KVM processes in the user context please re-patch your packages from the latest commits and reboot - there were some changes made specifically for it.

[axet]

It looks fine. Stand a night. Few additional suspend / resume. Looks like a fix! Thanks! Congratulations! Tricky one.

[n-peugnet]

@msizanoen1:

Can you check if applying #25662 fixes the problem? (You can find out how to do so by consulting your distribution's packaging documentation)

Just built it on Debian bookworm. From the resulted packages I installed systemd_252.2-1_amd64.deb, libsystemd0_252.2-1_amd64.deb and libsystemd-shared_252.2-1_amd64.deb then rebooted. I hope it's enough.

For now the issue didn't show up. I will keep you updated.

I also tested 252.2-1 from Debian's repos just before and I had the issue with the first suspend-then-hibernate.

[n-peugnet]

Still no sign of the issue, seems like it fixed it for me too.

[IamNaN]

Sorry to zombie this issue. My AMD Ryzen 7 7700X based motherboard won't resume from suspend. I've been using bookworm, fresh install, not an upgrade, since January. This issue just cropped up since the last update.

Additional note, suspend triggers the AMD RX 7900 XT GPU fans to jump to 100%. This is like the boot phase just before POST.

The situation occurs when timed out or manually suspended.

[msizanoen1]

Sorry to zombie this issue. My AMD Ryzen 7 7700X based motherboard won't resume from suspend. I've been using bookworm, fresh install, not an upgrade, since January. This issue just cropped up since the last update.

Additional note, suspend triggers the AMD RX 7900 XT GPU fans to jump to 100%. This is like the boot phase just before POST.

The situation occurs when timed out or manually suspended.

@IamNaN Can you run systemctl log-level debug prior to invoking systemctl suspend-then-hibernate, and then use journalctl -b > log.txt to generate a log file and upload it here?

[IamNaN]

@IamNaN Can you run systemctl log-level debug prior to invoking systemctl suspend-then-hibernate, and then use journalctl -b > log.txt to generate a log file and upload it here?

Thanks for looking. Sure thing. And another data point for you: this is a dual boot system. I have another drive I select from BIOS (no shared partitions or drives) with Windows 11, and that does recover from suspend. I don't think it's a hardware issue.

log.txt

[msizanoen1]

@IamNaN The log file doesn't contain the necessary information for troubleshooting. Please provide details on whether you're using suspend-then-hibernate or just normal suspend, and follow the following steps:

• Run sudo systemctl log-level debug.
• Run sudo systemctl suspend-then-hibernate.
• If the system fails to resume properly, hard reset the system then create a log file of the previous boot with sudo journalctl -b -1 > log1.txt.