session leaked due to missed inotify of cgroup becoming empty

[ddstreet]

systemd version the issue has been seen with

git

Used distribution

ubuntu

Expected behaviour you didn't see

all finished sessions are removed

Unexpected behaviour you saw

some sessions remained after all their processes exited and their cgroup was empty

Steps to reproduce the problem
To reproduce, the last process in a session's cgroup needs to exit while pid1 is performing a reexec or reload, during the period between removing its inotify watch on the 'cgroup.events' file, and adding the inotify watch again after the reexec/reload.

One way is to create a large number of sessions and then perform reloads while those sessions exit:

$ loginctl list-sessions
SESSION UID USER SEAT TTY
1 1000 ubuntu ttyS0

1 sessions listed.
$ for i in {1..100}; do sudo -b -i -u ubuntu ssh localhost -- sleep 1; done; for i in {1..20}; do echo 'Reloading...'; sudo systemctl daemon-reload; done
Reloading...
Reloading...
Reloading...
Reloading...
Reloading...
Reloading...
Reloading...
Reloading...
Reloading...
Reloading...
Reloading...
Reloading...
Reloading...
Reloading...
Reloading...
Reloading...
Reloading...
Reloading...
Reloading...
Reloading...
$ loginctl list-sessions
SESSION UID USER SEAT TTY
1 1000 ubuntu ttyS0
350 1000 ubuntu
351 1000 ubuntu
360 1000 ubuntu
373 1000 ubuntu
374 1000 ubuntu
375 1000 ubuntu
376 1000 ubuntu
380 1000 ubuntu
381 1000 ubuntu
386 1000 ubuntu
390 1000 ubuntu

many (but not all) sessions will have 'leaked'. Checking if the session cgroups are empty show they are:

$ grep populated /sys/fs/cgroup/unified/user.slice/user-1000.slice/session-*.scope/cgroup.events
/sys/fs/cgroup/unified/user.slice/user-1000.slice/session-1.scope/cgroup.events:populated 1
/sys/fs/cgroup/unified/user.slice/user-1000.slice/session-350.scope/cgroup.events:populated 0
/sys/fs/cgroup/unified/user.slice/user-1000.slice/session-351.scope/cgroup.events:populated 0
/sys/fs/cgroup/unified/user.slice/user-1000.slice/session-360.scope/cgroup.events:populated 0
/sys/fs/cgroup/unified/user.slice/user-1000.slice/session-373.scope/cgroup.events:populated 0
/sys/fs/cgroup/unified/user.slice/user-1000.slice/session-374.scope/cgroup.events:populated 0
/sys/fs/cgroup/unified/user.slice/user-1000.slice/session-375.scope/cgroup.events:populated 0
/sys/fs/cgroup/unified/user.slice/user-1000.slice/session-376.scope/cgroup.events:populated 0
/sys/fs/cgroup/unified/user.slice/user-1000.slice/session-380.scope/cgroup.events:populated 0
/sys/fs/cgroup/unified/user.slice/user-1000.slice/session-381.scope/cgroup.events:populated 0
/sys/fs/cgroup/unified/user.slice/user-1000.slice/session-386.scope/cgroup.events:populated 0
/sys/fs/cgroup/unified/user.slice/user-1000.slice/session-390.scope/cgroup.events:populated 0

Simply writing to them will bump their mtime and generate the missed inotify event, causing systemd to clear them all out:

$ echo '' | sudo tee /sys/fs/cgroup/unified/user.slice/user-1000.slice/session-*.scope/cgroup.events

tee: /sys/fs/cgroup/unified/user.slice/user-1000.slice/session-1.scope/cgroup.events: Invalid argument
tee: /sys/fs/cgroup/unified/user.slice/user-1000.slice/session-350.scope/cgroup.events: Invalid argument
tee: /sys/fs/cgroup/unified/user.slice/user-1000.slice/session-351.scope/cgroup.events: Invalid argument
tee: /sys/fs/cgroup/unified/user.slice/user-1000.slice/session-360.scope/cgroup.events: Invalid argument
tee: /sys/fs/cgroup/unified/user.slice/user-1000.slice/session-373.scope/cgroup.events: Invalid argument
tee: /sys/fs/cgroup/unified/user.slice/user-1000.slice/session-374.scope/cgroup.events: Invalid argument
tee: /sys/fs/cgroup/unified/user.slice/user-1000.slice/session-375.scope/cgroup.events: Invalid argument
tee: /sys/fs/cgroup/unified/user.slice/user-1000.slice/session-376.scope/cgroup.events: Invalid argument
tee: /sys/fs/cgroup/unified/user.slice/user-1000.slice/session-380.scope/cgroup.events: Invalid argument
tee: /sys/fs/cgroup/unified/user.slice/user-1000.slice/session-381.scope/cgroup.events: No such device
tee: /sys/fs/cgroup/unified/user.slice/user-1000.slice/session-386.scope/cgroup.events: No such device
tee: /sys/fs/cgroup/unified/user.slice/user-1000.slice/session-390.scope/cgroup.events: No such device
$ loginctl list-sessions
SESSION UID USER SEAT TTY
1 1000 ubuntu ttyS0

1 sessions listed.

[mbiebl]

Might be related to #20136