Systemd kills user process after logout even if KillUserProcesses is set to no

[Skarlso]

Submission type

• Bug report

systemd version the issue has been seen with

ubuntu@hostname:~$ systemctl --version
systemd 234
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN default-hierarchy=hybrid

In case of bug report: Expected behaviour you didn't see

I expect the user process to remain.

In case of bug report: Unexpected behaviour you saw

The process is still killed.

In case of bug report: Steps to reproduce the problem

I have a user process defined like this:

ubuntu@ip-172-31-56-155:~$ cat .config/systemd/user/my_service.service
[Unit]
Description=Test Service

[Service]
Type=simple
WorkingDirectory=/home/ubuntu
ExecStart=/home/ubuntu/test_service.sh
Restart=always
RemainAfterExit=yes
KillMode=none

[Install]
WantedBy=default.target

The file in question does this:

ubuntu@ip-172-31-56-155:~$ cat test_service.sh
#!/usr/bin/env bash

while true
do
    echo "
 ______
< `date` >
 ------
"
    sleep 1
done

I have the setting in logind.conf:

[Login]
#NAutoVTs=6
#ReserveVT=6
KillUserProcesses=no
#KillOnlyUsers=
KillExcludeUsers=root ubuntu
#InhibitDelayMaxSec=5
#HandlePowerKey=poweroff
#HandleSuspendKey=suspend
#HandleHibernateKey=hibernate
#HandleLidSwitch=suspend
#HandleLidSwitchDocked=ignore
#PowerKeyIgnoreInhibited=no
#SuspendKeyIgnoreInhibited=no
#HibernateKeyIgnoreInhibited=no
#LidSwitchIgnoreInhibited=yes
#HoldoffTimeoutSec=30s
#IdleAction=ignore
#IdleActionSec=30min
#RuntimeDirectorySize=10%
#RemoveIPC=yes
#InhibitorsMax=8192
#SessionsMax=8192
#UserTasksMax=33%

I can confirm that the status of the service is running when I'm logging out.

Then when I log back in:

Mar 19 13:12:55 ip-172-31-56-155 systemd[1257]: Stopped Test Service.

And the service is no longer running.

I ran out of ideas. I tried upgrading the latest version too as shown by the version indicator. If I remain logged in the service runs fine. I logged in on a different terminal session so had two login prompts and logged out with one of them. The service remained running. Meaning it wasn't tied to a single user session.

Please advice. I would have thought settings these settings makes it so that when I log out my service remains running. And I did read a lot about this and this should absolutely be the behavior I can expect.

Thanks!

[sourcejedi]

I think you can't get anywhere useful with this example? To be clear: the systemd --user instance is shared between all the login sessions of the same user. When the user closes their last login session, the systemd --user instance is immediately asked to shut down.

If you want to always run your systemd --user instance, so you can run user services and timer units and stuff permanently, and immediately on boot, there is a setting for that: loginctl enable-linger.

There's some possibility for different settings than linger. See the PR #4434, for a proposed setting which doesn't start on boot, but lets systemd --user instances pile up and only get stopped when the entire system is shut down. The maintainers were looking to accept it, it just needs implementation changes.

Me, I have my own thought (see end :).

---

If you have a service process managed by systemd --user, when you shut down systemd --user, it will necessarily cease to be managed. So it suggests you didn't particularly want a systemd service in the first place.

Ignoring that, I don't think we can really re-adopt existing service processes when you next start systemd --user. I think the other alternative, trying to ignore the process when we next start, is also a problem. We'll start the service a second time; also the old service process will still be in the same cgroup, named after the user service. I think there are non-trivial problems if you try to implement either of these.

Right now once the systemd --user process is gone, everything else will get SIGKILL, because:

$ systemctl --system show -p KillMode user@1000.service
KillMode=mixed

This setting might also be the least surprising when you're trying to generalize from other services or units... it lets the admin request shutdown of the set of user processes, without having to re-check the status and then resort to systemctl kill.

---

OTOH. Ow. By the same point you can't start tmux / screen from inside gnome-terminal-server.service; it won't survive logout. Personally I don't see this as just a bug in tmux/screen; loginctl enable-linger is just too big a hammer, and I don't see the proposed start-on-login as much better. It feels to me what is really needed for tmux is something close to machinectl shell $USER@.host /usr/bin/tmux start-server, that systems can configure to allow without requiring privileges. Another way to look at it: this would be implemented by starting a transient system service unit ala systemd-run, with some PamName= setting on the transient service.

A lot of systems already allow this by default - with up to a minute's delay - it's called crontab. Seems it would require a daemon and logind would be the obvious choice to absorb the function.

[poettering]

KillUserProcesses= applies to processes that are part of the user session when the user logs out, it has no effect on the user's singleton "systemd --user" instance, that is shared by all sessions of a user, and is refcounted by the sessions.

Please use "loginctl set-linger" on your user to tell logind to keep the user instance around even when the user is not logged in, which is the behaviour you are looking for here

[Skarlso]

Thanks for the clarification guys. Much appreciated!

[Skarlso]

@sourcejedi Thank you for the lengthy explanation and taking to time to write it out. Things make much more sense now. 👍

[deftdawg]

I have the same issue, I tried modifying logind.conf and also running loginctl set-linger... however, the later fails with "unknown operation"... How can I stop systemd from killing my tmux, nohup, screen, etc?

[sourcejedi]

@deftdawg I think my comment is correct and I suspect @poettering made a typo. man loginctl says the command is loginctl enable-linger, and there is no command loginctl set-linger.

[TomFreudenberg]

Sorry but coming back in 2023.

I have issues using Debian 11,12 and Ubuntu 22.04 (

systemd 249 (249.11-0ubuntu3.11)
+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY -P11KIT -QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified

while Debian 10 and Ubuntu 20.04 work.

---

https://unix.stackexchange.com/questions/766227/debian-11-12-and-ubuntu-22-04-terminate-screen-root-processes-when-started-from

When starting processes due to screen via console, those get killed on logout.

login: root
password: ***

then starting some command:

screen -d -S sample -m bash -c '/hom/app/server start'

After this the service is running and screen shows that process

screen -ls

# output
# There are screens on:
#   1805.sample	(01/05/24 12:13:18)	(Detached)
# 1 Sockets in /run/screen/S-root.

But when I logout, also the screen session will be destroyed.

Doing the same via ssh on the same machine by root (with RootLoginPermit) evrything runs on also after logout.

I tested the same actions on Ubuntu 20.04 and 22.04.

It works with no issues on Ubuntu 20.04 also when started by console session.

But it doesn't work on Ubuntu 22.04 neither on console.

It is a problem on Debian (tested bullseye and bookworm with latest updates) and Ubuntu (tested 22.04).

All found solutions like KillUserProcesses=no are checked and set by default. I also installed a fresh new debian with the same behaviour.

---

On debian and ubuntu it is always the same screen version Screen version 4.08.00 (GNU) 05-Feb-20

---

UBUNTU

1. Doing a fresh install from Ubuntu 20.04
2. Running screen in detached mode
3. logout and login again
4. screen processes still there

5. Doing an Ubuntu release upgrade to Ubuntu 22.04
6. Running screen in detached mode
7. logout and login again
8. screen processes are killed

---

DEBIAN

1. Doing a fresh install from Debian 10
2. Running screen in detached mode
3. logout and login again
4. screen processes still there

5. Doing an Debian release upgrade to Debian 11
6. Running screen in detached mode
7. logout and login again
8. screen processes are killed

---

So there is some significant, but currently to me unknown, change in the releases.

[mbiebl]

Are you using gnome-terminal (or a similar terminal emulator) that uses a system --user service?

[TomFreudenberg]

No, as written above it is the console.

In my case I checked with Proxmox (CT) and Vmware and both times I used their web interfaces and start the console from that point. The device is always /dev/tty1

P.S. please be aware of, that it does work (no screen jobs get killed on logout) when I login from remote via ssh.

[arvidjaar]

it is the console

I cannot reproduce it with systemd 254.4 (openSUSE Tumbleweed). Ubuntu 22.04 is using systemd 249.11 (as of now), and I am not sure how you login as root in the first place.

But yes, with "modern" session management via systemd user service the KillUserProcesses became rather useless - GUI session simply does not have user processes that are covered by this setting.

[mbiebl]

I can not reproduce the problem with Debian sid/trixie/bookworm either when logging in on the Linux console. A screen session survices a logout.

[TomFreudenberg]

@mbiebl, what is your /dev/... in that case? Also /dev/tty1?

---

@arvidjaar I have no GUI elements installed - just console - I am running components like rabbitmq, postgresql, webserver on it - no frontend elements

I will check it out once more with the debian netinst image and give you additional feedback. Really crazy while it works on Debian 10 and Ubuntu 20.04 but not newers.

[mbiebl]

It was tty2 (since gdm runs on tty1). But it doesn't really make a difference. It also works with tty1 if I stop gdm