New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

QL aborts on session logout and shutdown because X server is not available anymore #2897

Open
marcelpaulo opened this Issue Jul 7, 2018 · 27 comments

Comments

Projects
None yet
2 participants
@marcelpaulo

marcelpaulo commented Jul 7, 2018

I run QL master ( 56b68e5) on Xubuntu 18.04 and it has been nagging me for a very long time that when I logout from my current session or shutdown, QL doesn't retain its current state (currently playing song, filter string, ...). Investigating issue #1562, I discovered that QL abruptly aborts without saving its state on session logout (and shutdown).

I ran quodlibet --debug capturing the output in 3 situations, and here are the shutdown messages:

  1. Quit QL (full log)

    D: 24.437: qltk.window.__restore_state: Restore state
    D: 24.438: qltk.window.__restore_position: Restore position
    D: 24.438: qltk.window.__restore_size: Restore size
    D: 24.440: main.before_quit: Saving active browser state
    D: 24.441: main.before_quit: Shutting down player device 'GStreamer: 1.14.1.0'.
    D: 24.452: PluginManager.save: Saving plugins: 8 active
    D: 24.452: PluginManager.enable: Disable 'Theme Switcher'
    D: 24.510: PluginManager.enable: Disable 'ToggleMenuBar'
    D: 24.511: PluginManager.enable: Disable 'mpris'
    D: 24.654: PluginManager.enable: Disable 'queue'
    D: 24.655: PluginManager.enable: Disable 'Website Search'
    D: 24.655: PluginManager.enable: Disable 'editplaycount'
    D: 24.655: PluginManager.enable: Disable 'CustomCommands'
    D: 24.655: PluginManager.enable: Disable 'include_saved'
    D: 24.656: _main.quit_gtk: Quit GTK: done.
    D: 25.340: _main.run: Gtk.main() done.
    D: 25.340: library.save: Saving all libraries...
    D: 25.341: SongFileLibrary.save(<libraries.SongFileLibrary object at 0x7feb4abdccf0 (quodlibet+library+libraries+SongFileLibrary at 0x29cf6c0)>): Saving contents to '/home/paulo/.config/quodlibet/songs'.
    D: 34.721: config.save: Writing config...
    D: 34.763: main.main: Finished shutdown.
    
  2. Logout (full log)

    Gdk-Message: 13:37:09.077: quodlibet: Fatal IO error 11 (Resource temporarily unavailable) on X server :0.0.
    
  3. Shutdown (full log)

    Gdk-Message: 13:38:22.261: quodlibet: Fatal IO error 11 (Resource temporarily unavailable) on X server :0.0.
    

All processes receive SIGHUP on logout (and shutdown), so the X server also shuts down on logout (and shutdown, obviously). When QL receives SIGHUP, I get the impression that it attempts some operation that access the X server, most likely calling some GTK function, but without catching errors, so it simply aborts, not saving its state.

So, somehow once QL starts to shutdown, it should catch errors when calling any GTK functions (or any other functions that access the X server). If that's not feasible, then it should save its state before running any such X-accessing commands.

@lazka

This comment has been minimized.

Member

lazka commented Jul 7, 2018

Are you using systemd?

@marcelpaulo

This comment has been minimized.

marcelpaulo commented Jul 7, 2018

No, I gave up on systemd because QL aborted just the same.

@lazka

This comment has been minimized.

Member

lazka commented Jul 7, 2018

I meant systemd in general, but I guess that means yes.

@marcelpaulo

This comment has been minimized.

marcelpaulo commented Jul 7, 2018

My bad, I should have been more explicit: I'm running QL from Xfce's Whisker Menu:
image
which uses the io.github.quodlibet.QuodLibet.desktop file to start QL.

@marcelpaulo

This comment has been minimized.

marcelpaulo commented Jul 7, 2018

Digging a bit more, I ran this script foo:

#!/bin/bash

exec &>>"$0.log"

trap 'echo "caught SIGHUP"; exit' HUP
trap 'echo "caught SIGTERM"; exit' TERM
trap 'echo exiting' EXIT

echo START
while true; do
    date +'%F %T'
    sleep 5
done

from the Whisker Menu, using this desktop file:

[Desktop Entry]
Type=Application
Name=foo
Exec=foo

And discovered that:

  1. On logout, the script doesn't receive any signals and remains running
  2. On shutdown, it receives SIGHUP

So:

  1. On logout, QL is not receiving any signals, but aborts when it tries to access the X server (which will have been stopped by lightdm, the display manager)
  2. On shutdown, QL does receive SIGHUP, but aborts when it tries to access the X server, before it has the chance of saving its state
@marcelpaulo

This comment has been minimized.

marcelpaulo commented Jul 7, 2018

Continuing to dig, I ran the same foo script with the most basic systemd user service ~/.config/systemd/user/foo.service:

[Unit]
Description=foo

[Service]
ExecStart=/home/paulo/stage/bin/foo

and discovered:

  1. On logout, no signals, script remains running
  2. On shutdown, SIGTERM
@marcelpaulo

This comment has been minimized.

marcelpaulo commented Jul 8, 2018

Continuing to dig … Have just discovered that systemd-logind, the systemd login manager, has a property, KillUserProcesses, that determines whether all user processes are killed on session logout. The default is no. I have created file /etc/systemd/logind.conf.d/10-KillUserProcesses.conf:

[Login]
KillUserProcesses=yes

and my foo script did receive SIGTERM when I logged out. This would guarantee that QL receives SIGTERM on logout and shutdown. But … this setting messes with tmux and screen, whose sessions are usually left lingering for reattachment later, and also stops any other long-running user processes meant to linger on. There are alternatives to solve these latter inconveniences, but I haven't got around to them yet.

@lazka

This comment has been minimized.

Member

lazka commented Jul 8, 2018

@marcelpaulo

This comment has been minimized.

marcelpaulo commented Jul 8, 2018

Thanks for that ! Issue reported in 2015, and not solved yet :-(. I do like Xfce, though.

It looks like systemd KillUserProcesses=yes is the way to go, with the added bonus that it's desktop-agnostic.

However … it doesn't fix QL shutdown sequence: even with KillUserProcesses=yes in place, it still aborts just the same, I have just tested it:

Gdk-Message: 00:51:36.589: quodlibet: Fatal IO error 0 (Success) on X server :0.0.

Full --debug output is here

@lazka

This comment has been minimized.

Member

lazka commented Jul 8, 2018

Thanks for that ! Issue reported in 2015, and not solved yet :-(. I do like Xfce, though.

2013

However … it doesn't fix QL shutdown sequence: even with KillUserProcesses=yes in place, it still aborts just the same, I have just tested it:

My guess is that whatever kills X does so first, without waiting for apps to quit.

@lazka

This comment has been minimized.

Member

lazka commented Jul 8, 2018

As noted in the bug report the dev version reimplements the gnome dbus api to register clients with the session manager. But it's unclear when 4.14 will be released.

@marcelpaulo

This comment has been minimized.

marcelpaulo commented Jul 8, 2018

My guess is that whatever kills X does so first, without waiting for apps to quit.

I confess I never quite grasped the activation/shutdown sequence of all the components of a typical desktop session, and how systemd fits in. I do know that X is started/stoped by the display manager, lightdm for Xubuntu, and lightdm is a systemd service.

Your comment made me remember something I thought this afternoon and later forgot: investigate a setup to start QL as a systemd service in a way that, on session logout, QL is stopped before X.

This has a big caveat: AFAIK, please correct me, QL shutdown in asynchronous. I've tried quodlibet --quit, and echoing quit on ~/.config/quodlibet/control, but they returned immediately while QL went on quitting asynchronously. For this setup to work, or even if Xfce is fixed on version 4.14, I imagine that it would need to stop QL synchronously. Is there a way to do that ?

@lazka

This comment has been minimized.

Member

lazka commented Jul 8, 2018

This has a big caveat: AFAIK, please correct me, QL shutdown in asynchronous. I've tried quodlibet --quit, and echoing quit on ~/.config/quodlibet/control, but they returned immediately while QL went on quitting asynchronously.

I don't think making the quit command sync would help in any way, it's used by the user, not the system.

For this setup to work, or even if Xfce is fixed on version 4.14, I imagine that it would need to stop QL synchronously. Is there a way to do that ?

You can make any async operation for quitting an app sync by waiting until the process is gone.

@marcelpaulo

This comment has been minimized.

marcelpaulo commented Jul 8, 2018

I don't think making the quit command sync would help in any way, it's used by the user, not the system

In this case, it would be used by the system: systemd would synchronously stop QL, and when it finished, it could stop X. Hypothetically, of course, since I don't know if it's feasible.

You can make any async operation for quitting an app sync by waiting until the process is gone

Yes, you're right. I was thinking of something QL-native, that didn't require polling for the process existence.

@lazka

This comment has been minimized.

Member

lazka commented Jul 8, 2018

ok, so three things to do here:

  • Implement the xfce dbus api
  • Implement X11 session management
  • Use the first available method: gnome -> xfce -> x11

@lazka lazka added the enhancement label Jul 8, 2018

@lazka

This comment has been minimized.

Member

lazka commented Jul 8, 2018

All the above is in master now. xfce still doesn't wait even though we are registered with the session manager now, but at least we get a signal and try saving.

@lazka

This comment has been minimized.

Member

lazka commented Jul 8, 2018

maybe related: #1141

@marcelpaulo

This comment has been minimized.

marcelpaulo commented Jul 8, 2018

Wow, great stuff ! It did make a big difference, there was almost time to save everything, here's QL shutting down on logout/shutdown:

D: 37.491: session.xsmp.save_yourself: xsmp: save_yourself (<SaveType.GLOBAL: 0>, True, <InteractStyle.ANY: 2>, False)
D: 37.534: session.xsmp.die: xsmp: die
D: 37.535: qltk.window.__restore_state: Restore state
D: 37.535: qltk.window.__restore_position: Restore position
D: 37.535: qltk.window.__restore_size: Restore size
D: 37.537: main.before_quit: Saving active browser state
D: 37.537: main.before_quit: Shutting down player device 'GStreamer: 1.14.1.0'.
D: 37.545: PluginManager.save: Saving plugins: 8 active
D: 37.545: PluginManager.enable: Disable 'Theme Switcher'
D: 37.618: PluginManager.enable: Disable 'ToggleMenuBar'
D: 37.618: PluginManager.enable: Disable 'mpris'
D: 37.760: PluginManager.enable: Disable 'queue'
D: 37.761: PluginManager.enable: Disable 'Website Search'
D: 37.761: PluginManager.enable: Disable 'editplaycount'
D: 37.762: PluginManager.enable: Disable 'CustomCommands'
D: 37.762: PluginManager.enable: Disable 'include_saved'
D: 37.763: _main.quit_gtk: Quit GTK: done.
D: 38.453: _main.run: Gtk.main() done.
D: 38.454: library.save: Saving all libraries...
D: 38.454: SongFileLibrary.save(<libraries.SongFileLibrary object at 0x7f30edcbbd80 (quodlibet+library+libraries+SongFileLibrary at 0x1e586c0)>): Saving contents to '/home/paulo/.config/quodlibet/songs'.

I'm intrigued by the registration with the session manager:

D: 19.170: session.init: Trying GnomeSessionClient
D: 19.171: GnomeSessionClient.open: Connecting with gnome session manager
D: 19.172: session.init: org.gnome.SessionManager unowned
D: 19.173: session.init: Trying XfceSessionClient
D: 19.173: XfceSessionClient.open: Connecting with gnome session manager
D: 19.175: session.init:
g-dbus-error-quark: GDBus.Error:org.freedesktop.DBus.Error.UnknownMethod: Method "RegisterClient" with signature "ss" on interface "org.Xfce.Session.Manager" doesn't exist
 (19)
D: 19.176: session.init: Trying XSMPSessionClient
D: 19.180: XSMPSessionClient.open: Connecting with XSMP
D: 19.181: XSMPSessionClient.open: Connected. Client ID: 2fc7d6cad-7089-44ae-bb9e-765eb3614536

Looking at the session bus with d-feet:
image

The interface name is org.xfce.Session.Manager and not org.Xfce.Session.Manager, but still it doesn't have any methods to register clients, even though it has ListClients, it emits signal ClientRegistered, and there are registered clients:
image

These are the registered clients:

pulseaudio
xfwm4
xfce4-panel
xfce4-power-manager
xfdesktop
Thunar
xfsettingsd
xfce4-terminal

I wonder how these clientes registered.

I couldn't find any traces of the X11 session manager on neither of the d-bus buses (system and session). It seems magical how it works.

maybe related: #1141

Directly related.

@marcelpaulo

This comment has been minimized.

marcelpaulo commented Jul 8, 2018

Most important of all: thank you so much, @lazka !

@lazka

This comment has been minimized.

Member

lazka commented Jul 9, 2018

The interface name is org.xfce.Session.Manager and not org.Xfce.Session.Manager, but still it doesn't have any methods to register clients, even though it has ListClients, it emits signal ClientRegistered, and there are registered clients:

Oops, thanks. I copied it from the docs, but it's wrong there. The missing method is expected and should be there with 4.14: https://bugzilla.xfce.org/show_bug.cgi?id=10295#c5

@marcelpaulo

This comment has been minimized.

marcelpaulo commented Jul 9, 2018

xfce still doesn't wait even though we are registered with the session manager now

I think I finally understood what you meant. The end of session protocol:

https://git.xfce.org/xfce/xfce4-session/commit/?id=04810a0b4d2664b3ada9ad0f3c58b3536cf365f4

is not implemented in the current xfce version, 4.12.1.

I'm still a bit confused about who kills what and when. I thought systemd was the sole killer, but thanks to you, I learned now that the session manager is also a killer. I wonder when each one steps in killing processes. I changed my foo script to sleep for 10s and exit when it catches SIGTERM, and set up a systemd service with ExecStop thats kills it synchronously, and TimeoutStopSec=15. So, as far as systemd is concerned, on logout it will run ExecStop and wait for 15s for the script to end. I see in the journal that after about 5s, the script just dies, so I presume someone sent it SIGKILL. Perhaps it was the xfce session manager, this grim reaper !

@marcelpaulo

This comment has been minimized.

marcelpaulo commented Jul 9, 2018

Discovered a couple of things about xfce4-session-manager, the grim reaper:

  1. Setting environment variable XFSM_VERBOSE makes it log verbosely in ~/.xfce4-session.verbose-log. When a new session starts, it rotates the previous log file to ~/.xfce4-session.verbose-log.last. Here's ~/.xfce4-session.verbose-log.last, after running xfce4-session-logout --logout. The clients seems to use some sort of IPC to register themselves, and there is a shutdown protocol with 2 phases which seems the same as in https://git.xfce.org/xfce/xfce4-session/commit/?id=04810a0b4d2664b3ada9ad0f3c58b3536cf365f4
  2. When logging out with loginctl terminate-session or systemctl restart lightdm, systemd runs ExecStop on its user services and sends SIGTERM to the non-service processes, including xfce4-session-manager and all other xfce processes
  3. When logging out with xfce4-session-logout --logout, systemd does the same as in (2), but there are no xfce processes, which have already terminated, as the xfce4-session-manager log indicates
  4. Both in (2) and (3), processes which have not died within about 5s seem to receive SIGKILL (I saw that my foo systemd user service aborts after 5s) I presume that xfce4-session-manager sends SIGKILL to the session control group after it finishes handling all registered clients.

From (1), it looks like xfce4-session-manager waits for its registered clients to end. Could QL use that non-dbus interface to register as a client ?

@marcelpaulo

This comment has been minimized.

marcelpaulo commented Jul 9, 2018

According to xfce4-session git master:

4.13.0
======
- xfsm has been ported to Gtk3.
- xfsm now does session management for clients using the Gtk application
  dbus interface.
- xfsm has been ported to GDBus.

Build fails, a header file is missing, I've opened a bug report. Not sure if it's possible to run xfce4-session-manager 4.13.0 while all oher xfce4 components are at 4.12, I'll try once I manage to build it.

EDIT: An afterthought: unless xfce4-session-manager 4.13.0 still supports 4.12 client interaction, all current 4.12 clients will fail to register.

@lazka

This comment has been minimized.

Member

lazka commented Jul 9, 2018

From (1), it looks like xfce4-session-manager waits for its registered clients to end. Could QL use that non-dbus interface to register as a client ?

It waits for the save request response which comes before the one indicating a shutdown. If QL splits the shutdown part and the saving part we could save there and shutdown would then just be a process exist. Currently both are intermixed..

Definitely something we should look into in the long run.

@marcelpaulo

This comment has been minimized.

marcelpaulo commented Jul 9, 2018

Hey, really good to know a definitive solution is feasible.

@marcelpaulo

This comment has been minimized.

marcelpaulo commented Jul 10, 2018

A rather crude but simple workaround: I whipped up 2 scripts, one that kills a process synchronously given a regex to match its full command line, and another that calls the first script to kill quodlibet synchronously and then call systemd or loginctl to run an action: poweroff, reboot or logout. I then bound this script to key sequences: ctrl-alt-end for poweroff, ctrl+alt+delete for logout, and ctrl+alt+backspace for reboot.

This will do while xfce's session manager doesn't implement the D-Bus interface and QL doesn't change its shutdown sequence.

@marcelpaulo

This comment has been minimized.

marcelpaulo commented Jul 15, 2018

Just to leave it documented for reference, I found out a handy way of synchronously stopping QL with SIGTERM:

killall -wv quodlibet

killall takes a command name, which QL sets to quodlibet, -w makes killall to wait for QL to die, -v just makes it show a message.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment