[Arch Linux] Shutting down a VM hangs while consuming CPU

[ptitdoc]

Qubes OS version (e.g., R3.2):

R3.2

Affected TemplateVMs (e.g., fedora-23, if applicable):

archlinux

---

Expected behavior:

Shutting down the VM should be fast and quiet

Actual behavior:

Shutting down the VM hangs for about 1 minute with a Process running at 100% (25% CPU consumption).

Steps to reproduce the behavior:

Conditions to reproduce the behavior are unknown, but affected users are observing this each time their archlinux appvm shutdown.

General notes:

It seems to hangs until the user session timeout.
[ *] A stop job is running for Session 1 of user user (8s / 1min 30s)

---

Related issues:

N/A

[ptitdoc]

In my case, the problem can be solved by setting kernel.core_pattern=core
As referenced here:
systemd/systemd#1615

@benjarobin has done a very good job at explaining the two failures in a series of comments.
(See #2691 (comment) and following).

Both cases are linked with the application reacting badly to the SIGTERM sent to to them when the system goes down.

In the first case, the application crashes, but a bug with coredumps makes the system wait forever on the coredump service that can not start during shutdown. See #2993 for an attempt at fixing it.
In the second case, the application ignores (or misses) the SIGTERM signal and continues running, unaware of the shutdown.
As systemd cannot tell this situation from a service requiring more time to exit cleanly, it must wait untill the hard limit of 90s before killing the process with SIGKILL. Fixing the application is the way to go. For conky I have submitted brndnmtthws/conky#233.

Of course, this does not explain why so many programs started to fail with systemd versions around 229-3. I think one explanation might be that systemd started to send SIGHUP on top of SIGTERM (as outlined in https://bbs.archlinux.org/viewtopic.php?pid=1617469#p1617469).

[ptitdoc]

In my case, the problem can be solved by setting kernel.core_pattern=core
As referenced here:
systemd/systemd#1615

@benjarobin has done a very good job at explaining the two failures in a series of comments.
(See #2691 (comment) and following).

Both cases are linked with the application reacting badly to the SIGTERM sent to to them when the system goes down.

In the first case, the application crashes, but a bug with coredumps makes the system wait forever on the coredump service that can not start during shutdown. See #2993 for an attempt at fixing it.
In the second case, the application ignores (or misses) the SIGTERM signal and continues running, unaware of the shutdown.
As systemd cannot tell this situation from a service requiring more time to exit cleanly, it must wait untill the hard limit of 90s before killing the process with SIGKILL. Fixing the application is the way to go. For conky I have submitted brndnmtthws/conky#233.

Of course, this does not explain why so many programs started to fail with systemd versions around 229-3. I think one explanation might be that systemd started to send SIGHUP on top of SIGTERM (as outlined in https://bbs.archlinux.org/viewtopic.php?pid=1617469#p1617469).

[ptitdoc]

Coredump by default is set to |/bin/false. This apparently cause coredump hanging because an application crashes when systemd tries to kill it.
I don't know how to identify the application crashing during shutdown.

[ptitdoc]

Coredump by default is set to |/bin/false. This apparently cause coredump hanging because an application crashes when systemd tries to kill it.
I don't know how to identify the application crashing during shutdown.

[adrelanos]

ptitdoc:

Coredump by default is set to `|/bin/false`. This apparently cause coredump hanging because an application crashes when systemd tries to kill it. I don't know how to identify the application crashing during shutdown.

Perhaps enable persistent systemd journal log? sudo mkdir /var/log/journal Other changes may be required depending on that distribution. Does that help?

[adrelanos]

ptitdoc:

Coredump by default is set to `|/bin/false`. This apparently cause coredump hanging because an application crashes when systemd tries to kill it. I don't know how to identify the application crashing during shutdown.

Perhaps enable persistent systemd journal log? sudo mkdir /var/log/journal Other changes may be required depending on that distribution. Does that help?

[ptitdoc]

Potentially related to :
https://groups.google.com/forum/#!searchin/qubes-users/Arch$20Linux$20$2B$20i3%5D$20High$20CPU$20usage$20after$20closing$20windows%7Csort:relevance/qubes-users/pe-v7bLSRx4/Zk0v-vkSBwAJ

[ptitdoc]

Potentially related to :
https://groups.google.com/forum/#!searchin/qubes-users/Arch$20Linux$20$2B$20i3%5D$20High$20CPU$20usage$20after$20closing$20windows%7Csort:relevance/qubes-users/pe-v7bLSRx4/Zk0v-vkSBwAJ

[ptitdoc]

I tested updating to qubes-vm-core 3.2.18 and qubes-vm-gui 3.2.17 and I cannot observe this issue anymore.

[ptitdoc]

I tested updating to qubes-vm-core 3.2.18 and qubes-vm-gui 3.2.17 and I cannot observe this issue anymore.