Join GitHub today
GitHub is home to over 20 million developers working together to host and review code, manage projects, and build software together.
Regression in systemd-coredump 229: system hangs for 90 seconds during shutdown #2691
Comments
ohsix
commented
Feb 22, 2016
|
thanks for getting log first, lots of people are noticing this [ 114.155486] systemd[1]: session-c1.scope: Stopping timed out. Killing. this is the notable bit; something in your user session isn't being polite everything between |
diegoviola
commented
Feb 22, 2016
|
I've been experiencing the same problem since systemd 228 or earliest versions, I'm on Arch Linux (x86-64). Here's my logs: http://ix.io/owz |
|
@diegoviola , please file a different issue. The description of this issue says it's a regression of 229, so if you already saw this on 228 your problem must be different. |
|
There is not a lot of useful information in that log, I'm afraid. Please start the debug shell, issue the shutdown, then switch to the debug shell (Ctrl+Alt+F9) and check with |
martinpitt
added
needs-reporter-feedback
user-session
labels
Feb 22, 2016
diegoviola
commented
Feb 22, 2016
|
@martinpitt just because he didn't see the problem in 228 doesn't mean it's not the same issue, I've experienced this problem on Arch Linux as well as on Ubuntu. Also, when I was testing this yesterday (on Arch Linux), I booted my computer normally, I login from the tty and the only thing I had running was bash (no X or any other programs), then I typed I would be happy to re-open or provide a different bug report after this problem is fixed, and in case my problem still persist after fixing this issue, but I believe we're talking about the same problem here. |
snckrz
commented
Feb 22, 2016
|
Switching to the debug shell is not possible during the hanging. It doesn't do anything. (It does work during the session, so it is running correctly) |
|
This is the output of the pgrep command:
The pkill command did not work unless I added the -9 option, to issue SIGKILL. |
|
This seems to make the problem go away:
|
sjuxax
commented
Feb 23, 2016
|
Been seeing the same thing here on ArchLinux for the last couple of months. Assumed it was something I broke. |
|
Also see https://bugzilla.redhat.com/show_bug.cgi?id=1310608#c3. The problem is that there is a crash during shutdown and systemd-coredump takes awful lot of time to generate the dump. |
jurf
commented
Feb 23, 2016
|
I have the same symptoms but I'm not sure I'm experiencing the same thing. For me |
ohsix
commented
Feb 23, 2016
|
@diegoviola there's a thing with bugs and bug trackers, people are cooperating to try and find out what the bug actually is, you don't just jump on and say you have the same bug ... you can't unless you've already found the bug. please verify what people have seen in following comments to see if this really is the same issue. |
|
Although the problem seemed to go away for me by disabling systemd-coredump, there doesn't actually seem to be a systemd-coredump process running during the hang. These were all the processes without
|
|
Well, I've been experiencing the issue since at least systemd-227. It's not the coredump that's the problem. It's systemd user session (started by display manager via pam_systemd.so) that is blocking the shutdown. When the desktop session manager terminates Xorg and logs out the user, there are still services started by systemd user session (user bus and others that don't need X running). However, the session manager keeps running like the session is still there, even after the user has logged out. It's like the pam_systemd never told systemd logind that the user has logged out and it needs to terminate the user session. I observed the behaviour by setting KillUserProcesses=yes in logind.conf. Even when you log in from tty and then you log out, the systemd user daemon (and, depending on the session, dbus services) will remain running for some time, which is why shutdown hangs - it's waiting for user service to terminate on its own. It's like the PAM module never sends a "user logged out" notification to the session manager. |
|
Possible duplicate of issue #1615. |
gzmorell
commented
Mar 21, 2016
|
Hi, |
|
I made a mistake in my previous test, that why I deleted my previous message. With In KDE, when closing the session, plasma crash most of the time which lead to this session status :
The blocking process is |
kyawthusoe3142
commented
Mar 22, 2016
|
Same problem in Debian 8 systemd version 215-17. |
heavyhdx
commented
Mar 24, 2016
|
Still getting this issue on a fresh Arch+KDE installation. |
bmpro
commented
Mar 28, 2016
|
I can confirm. Same issue with 229. Reverted to 228 for now which works. |
firekage
commented
Mar 28, 2016
|
I have the same behaviour on my desktop machine and on netbook and laptop machines. Also, with a clean install. In my case i have "a stop job is running for user", "a stop job is running for c2 session", "a stop job is running for dile manager". |
firekage
commented
Mar 28, 2016
|
I have the same behaviour on my desktop machine and on netbook and laptop machines. Also, with a clean install. In my case i have "a stop job is running for user", "a stop job is running for c2 session", "a stop job is running for dile manager". Forget to mention: "a stop job is running for user manager UID 1000". |
xxxperimental
commented
Mar 28, 2016
|
Alright, but how do I downgrade systemd to 228?
Other than that has anyone found any other solutions to this issue? |
diegoviola
commented
Mar 28, 2016
|
How about git-bisect rather than downgrade? Downgrade is never a solution. |
|
The true source of this "bug" is very very old, I think it was always there. But the "true" source of this bug can be triggered with this simple program:
Boot the PC without any graphical login , login to a tty, launch this app Systemd wait 90s before sending SIGKILL (which is kind of expected, but 90s is quite long) |
|
To reproduce the issue with systemd-coredump, use this program :
Boot the PC without any graphical login , login to a tty, launch this app With v228 the PC poweroff without any problem, and with v229 the PC wait 90s... |
|
And I got the following result from the early shell (tty9) after running the application of my previous message (which wait the SIGTERM and crash with a segfault when the SIGTERM is received)
There are couple of thing very strange :
Ok, I get the associated log, and I think I get it. These lines are all from the same timestamp and this is the result of the systemctl poweroff
Here my thought :
|
benjarobin
referenced this issue
Mar 28, 2016
Closed
System hangs at reboot/shutdown - Arch Linux #1615
xxxperimental
commented
Mar 29, 2016
|
Would it be safe to somehow disable or avoid systemd-coredump, so this issue doesn't come up? |
|
Well I still have issues even if coredump is disabled...
I will try to investigate a little bit more this end of day. |
xxxperimental
commented
Mar 29, 2016
|
Perhaps there is a way to make it time-out and terminate whatever processes are still running? |
nabildanial
commented
Mar 29, 2016
|
I have this problem only when I am rebooting/shutdown from GNOME Flashback (Compiz). The hanging doesn't occur in GNOME 3. Arch Linux. |
|
I've bisected the bug to the following commit:
This change, which was apparently merged only 24 hours before the final release of systemd 229 (!!!), made systemd-coredump work in two parts: one part directly invoked by the kernel, and one part activated by a UNIX domain socket. For reasons that are not yet entirely clear to me, systemd refuses to start the socket-activated process during the shutdown sequence, and the crashing process hangs waiting for the coredump to complete. See, for example, the following log messages:
The bug can be reproduced by starting any program that crashes when it receives SIGTERM, such as the example program that benjarobin posted, then trying to reboot. The bug can be worked around by creating a file /etc/sysctl.d/50-coredump.conf with the following contents:
That causes the kernel to write coredumps directly, bypassing the buggy systemd code. It also should be noted that some people may be encountering 90-second hangs for other reasons, such as processes ignoring SIGTERM. The bug I have been encountering, though, has been caused specifically by the bug in systemd-coredump described above, introduced in systemd 229. @poettering, any thoughts on fixing this? |
ebiggers
changed the title from
Regression in systemd 229: system hangs for 90 seconds during shutdown
to
Regression in systemd-coredump 229: system hangs for 90 seconds during shutdown
Mar 30, 2016
|
@ebiggers Well ... I have already explained (#2691 (comment)) why there is a problem with systemd-core dump. |
Potomac
commented
Mar 30, 2016
|
the workaround found by ebiggers ( /etc/sysctl.d/50-coredump.conf file ) works, I no longer have the message "A stop job is running for Session c2 for user...." I use archlinux, systemd 229-3 |
|
I highly recommend not to do that, you may have random core file created on your filesystem.
|
Potomac
commented
Mar 30, 2016
|
@benjarobin : we can set a custom path for storing cores files, with the parameter kernel.core_pattern : http://unix.stackexchange.com/questions/192716/how-to-set-the-core-dump-file-location-and-name for example in /etc/sysctl.d/50-coredump.conf : %e is the filename |
Potomac
commented
Apr 4, 2016
|
despite the workaround found by Ebiggers I still have the bug ( "a stop job for session C2 user" ) it's a random bug, for example it can occur after 10 reboots ( that's why I thought it was solved, we need to test with a lot of reboot/shutdown, the bug will be considered as solved only when no bug occurs after at least 10~15 reboots ) |
added a commit
to ebiggers/systemd
that referenced
this issue
Apr 8, 2016
added a commit
to ebiggers/systemd
that referenced
this issue
Apr 8, 2016
ebiggers
referenced this issue
Apr 8, 2016
Closed
coredump: allow systemd-coredump to run during shutdown #2993
|
Well, seeing as the systemd maintainers have been unresponsive to this bug, despite it being open six weeks, having the most comments of any open bug, and being a regression in the latest release, I have attempted to fix it myself. See pull request #2993. The fix relaxes the dependencies of systemd-coredump@.service, allowing it to be started during the shutdown sequence. This fixes the problem for me. However, I do not know that it is the best solution; an alternative might be to disable coredumps during shutdown. Note that as I mentioned before, not everyone who may be encountering a 90 second wait during shutdown is necessarily encountering this exact bug with systemd-coredump. The 90 second wait is apparently something that happens whenever processes are unresponsive to SIGTERM. |
omergoktas
commented
Apr 17, 2016
it worked, thanks |
poettering
added this to the v230 milestone
Apr 18, 2016
poettering
added
bug
coredump
and removed
user-session
needs-reporter-feedback
labels
Apr 18, 2016
diegoviola
commented
Apr 23, 2016
|
Weird, I switched back to Firefox (from Chromium) and the issue went away. |
added a commit
to ebiggers/systemd
that referenced
this issue
Apr 23, 2016
added a commit
to evverx/systemd
that referenced
this issue
Apr 24, 2016
evverx
referenced this issue
Apr 24, 2016
Merged
tests: add test for https://github.com/systemd/systemd/issues/2691 #3101
added a commit
that referenced
this issue
Apr 25, 2016
added a commit
to poettering/systemd
that referenced
this issue
Apr 29, 2016
poettering
added
the
has-pr
label
Apr 29, 2016
|
My proposed fix is now waiting in #3148. Please have a look! |
added a commit
to poettering/systemd
that referenced
this issue
Apr 29, 2016
|
#3148 was applied now. Still would like to have some feedback if it fixes the coredump issues though. Will close this issue for now, but please report back should you still encounter problems even with this patch applied! thanks! |
poettering
closed this
Apr 29, 2016
|
Thanks Lennart! I just tested the latest systemd master, and the problem no longer occurs. The systemd-coredump service fails as before, but it does not impede shutdown. |
|
@ebiggers thanks for testing! And sorry this took so long! |
diegoviola
commented
May 1, 2016
|
Thanks @poettering |
ebiggers commentedFeb 22, 2016
I am occasionally (> 50% of the time) experiencing a hang during shutdown. The problem started after upgrading to systemd 229. Reverting to systemd 228 makes the problem go away.
Attached the shutdown-log.txt file generated as detailed at this page: https://freedesktop.org/wiki/Software/systemd/Debugging/
shutdown-log.txt