tmux server terminates when client via SSH closes its Terminal

[moritzdietz]

Issue description

I'd like to ask for some help troubleshooting this issue, as I can't find out what's causing this on my own.

I have the following workflow:

flowchart LR
    hosta(Host A)
    hostb(Host B)
    tmuxserver(tmux server)
    tmuxsession(tmux session)
    subgraph Host A local
        direction LR
        hosta --> Alacritty
        Alacritty --> SSH
    end
    subgraph Host B local
        direction LR
        SSH .-> hostb
        hostb --> tmuxserver
        tmuxserver --> tmuxsession
    end

Loading

This usually works pretty reliably. When Host A reboots or powers off, I can just resume my tmux session later on Host B.
Now, for a couple of days now but I can't pinpoint it to when exactly, I noticed that the tmux server would die on Host B whenever I close the Alacritty window and thus the SSH connection on Host A when I'm still inside of any tmux session on Host B.
This is always reproducible.

To troubleshoot I tried this:

flowchart LR
    subgraph Host B local
    direction LR
        hostb(Host B)
        tmuxserver(tmux server)
        tmuxsession(tmux session)
        hostb --> tmuxserver
        tmuxserver --> tmuxsession
    end

Loading

Open Alacritty locally on Host B with no tmux server running. I start tmux and thus a new server and session, then close Alacritty. Re-open Alacritty and run tmux ls - my session is still there. I can just re-attach to it as expected.

This seems to only occur whenever Host A connects to Host B via SSH and I close the Alacritty window on Host A or any other terminal like GNOME Terminal.
This issue does not occur when I detach from tmux via the tmux detach command. Then the server lives on.
It also occurs when you detach from the tmux session, close the SSH connection from Host A to Host B and re-connect via SSH; the tmux server died.

In the debug logs of the server I see this:

1656510343.992709 server_signal: Hangup
1656510343.992719 server_signal: Terminated
1656510343.992723 session 0 destroyed (server_send_exit)

Required information

Please provide the following information:

• tmux version (tmux -V).
  tmux build on: 2022-06-29 11:22:46+02:00 Commit: d8c527a5
• Platform (uname -sp).
  Linux x86_64
• $TERM inside and outside of tmux (echo $TERM).
  alacritty (outside), tmux-256color (inside)
• Logs from tmux (tmux kill-server; tmux -vv new).
  I have the debug logs and can submit them on request as this has sensitive environment variables.

[nicm]

1656510343.992719 server_signal: Terminated

This means something is sending your tmux server SIGTERM. Are you sure this isn't systemd being stupid? That's usually what causes this kind of thing.

It's a bit funny it doesn't happen when you detach tmux manually so I guess it is tied up somehow with ssh exiting.

[moritzdietz]

Oh, I just retested the detaching from the tmux session - it actually does kill the server! That's very odd.

journalctl says the following at that epoch timestamp:

Jun 29 15:45:43 lx0013 sshd[21024]: Received disconnect from X port 49084:11: disconnected by user
Jun 29 15:45:43 lx0013 sshd[21024]: Disconnected from user moritz X port 49084
Jun 29 15:45:43 lx0013 sshd[20971]: pam_unix(sshd:session): session closed for user moritz
Jun 29 15:45:43 lx0013 systemd[1]: Stopping Session 41 of user moritz.
Jun 29 15:45:43 lx0013 systemd[1]: session-41.scope: Succeeded.
Jun 29 15:45:43 lx0013 systemd[1]: Stopped Session 41 of user moritz.
Jun 29 15:45:43 lx0013 systemd-logind[708]: Removed session 41.

[moritzdietz]

I will cross test this on two totally different systems using the same tmux version to see if I can reproduce it somewhere else.

[moritzdietz]

Meh accidentally closed it…

[nicm]

I bet this is systemd, I can't remember what you need to change to fix this but there is a way.

[moritzdietz]

Yes, you were right. It is systemd. I found this https://unix.stackexchange.com/questions/490267/prevent-logoff-from-killing-tmux-session and it worked.
SSH into Host B and ran systemd-run --scope --user tmux, then closed Alacritty and re-opened it and SSHed into Host B -> the session is still there. Will eventually fix it in the systemd configuration but for now I have a workaround.

It now makes more sense to me as I re-installed that VM just recently -> I never had an issue on there before.

Thanks for helping me solve this! :)

[moritzdietz]

Just for completeness and for others coming across this issue here:

I just figured it out: The file /etc/systemd/logind.conf was altered on the VM Host B.
It had the this setting configured:

KillUserProcesses=yes

the systemd default value for this setting is no. I changed the setting to match its default value, rebooted the VM and the problem was gone.
https://www.freedesktop.org/software/systemd/man/logind.conf.html#KillUserProcesses=

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.