libvirtd socket activation not working since 253.5

[lkrms]

systemd version the issue has been seen with

253.5

Used distribution

Arch Linux (rolling)

Linux kernel version used

6.3.5-arch1-1

CPU architectures issue was seen on

x86_64

Component

systemd

Expected behaviour you didn't see

In its default configuration, libvirtd.service starts a libvirtd process that times out after 120 seconds of inactivity (no running domains and no connected clients). The service remains active and libvirtd is respawned on-demand via systemd socket activation (libvirtd-ro.socket, libvirtd.socket and libvirtd-admin.socket correspond to /var/lib/libvirt/{libvirt-sock-ro,libvirt-sock,libvirt-admin-sock}` respectively by default).

Unexpected behaviour you saw

After updating from systemd 253.4 to 253.5, socket activation stopped working. Attempting to connect to libvirtd via a socket after it times out yields nothing. Client software (I'm using virsh and virt-manager on the same host) halts indefinitely as soon as a libvirtd-dependent command is issued, and no errors are reported or logged before or after they are terminated (e.g. with Ctrl+C), including by systemd.

If systemctl restart libvirtd.service is issued in a separate terminal, libvirtd starts as normal and any processes trying to connect via a socket succeed immediately (there is no need to cancel the attempt and try again).

After rolling back systemd, libvirt and other updates installed at the same time in various combinations, the culprit appears to be systemd 253.5. Downgrading to 253.4 restores the desired behaviour.

Steps to reproduce the problem

• Install a libvirt package with stock systemd unit files
• Start libvirtd and/or enable its sockets as needed
• Wait for the libvirtd process to terminate after 2 minutes
• Run sudo virsh list or similar
• Wait indefinitely

Additional program output to the terminal or log subsystem illustrating the issue

Here's output from strace, in the absence of anything appearing in a log entry:

https://gist.github.com/lkrms/5bb9ef2e7ad8aa5d89e164b6e3f4d4a6#file-gistfile1-txt

[poettering]

This is not a useful bug report. Always provide debug logs that show what's going on, i.e. "systemd-analyze log-level debug" and then reproduce the issue and provide what the service manager (i.e. pid1) logged as result of the activation request.

what makes you think this is a systemd issue? where's this tracked at libvirt project?

[poettering]

After rolling back systemd, libvirt and other updates installed at the same time in various combinations, the culprit appears to be systemd 253.5. Downgrading to 253.4 restores the desired behaviour.

Uh, stable dot releases are not tracked in this repo, that's a systemd-stable git repo thing.

Can you reproduce this on current git main?

[Toolybird]

Bisected to ff32060 (Confirmed. Reverting this single commit makes it work.)
Not sure why this causes it... I haven't tried current git main yet...

Upstream libvirt issue here [1], but I believe it's barking up the wrong tree and contains misinformation (including from me).

libvirt daemons are documented here [2]. I am using the libvirt monolithic daemon.

[1] https://gitlab.com/libvirt/libvirt/-/issues/483
[2] https://libvirt.org/daemons.html

[Toolybird]

BTW, when libvirtd is working properly, this is often found in the journal which could well be related (it's been like this for years IIRC):

systemd[1]: libvirtd.service: Found left-over process 1125 (dnsmasq) in control group while starting unit. Ignoring.
systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
systemd[1]: libvirtd.service: Found left-over process 1126 (dnsmasq) in control group while starting unit. Ignoring.
systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.

the dnsmasq processes are definitely still running when the libvirtd.service shuts down...

[berrange]

the dnsmasq processes are definitely still running when the libvirtd.service shuts down...

Libvirt intentionally leaves various things running when it shuts down, because it needs to be able to stop & start (for software upgrades) without disrupting running guests. Hence it sets KillMode=process (as opposd to cgroup)

[poettering]

if libvirt watns to leave stuff running, it should probably move that stuff to a distinct scope (or service), so that it is unaffected by libvirt restarts.

[Toolybird]

Fedora Rawhide also impacted:

https://bugzilla.redhat.com/show_bug.cgi?id=2213660

/cc @msekletar

[MichaelRiss]

What is the intended behavior of systemd? When is systemd supposed to change the status of a service from "Active" to "Inactive"?
Is it
A: when the process with the Main PID exits?
or
B: when all processes in the CGroup have exited?

The commit ff32060 seems to flip the behavior from A to B. As libvirt depends on behavior A it worked so far and now as the behavior has changed to B libvirt fails. So the question for me here is: Is behavior B intended? If yes, then libvirt needs to be adapted. If not, it's a systemd bug.

[MrDisguised]

Hi guys!! To temporarily fix the issue follow the guide that I have mentioned in the arch forum - https://bbs.archlinux.org/viewtopic.php?pid=2104316#p2104316

[poettering]

What is the intended behavior of systemd? When is systemd supposed to change the status of a service from "Active" to "Inactive"? Is it A: when the process with the Main PID exits? or B: when all processes in the CGroup have exited?

This depends on ExitType=. It should be when the main process goes down (ExitType=main, the deault), or when the last process in the service's cgroup goes down (ExitType=cgroup).

The commit ff32060 seems to flip the behavior from A to B. As libvirt depends on behavior A it worked so far and now as the behavior has changed to B libvirt fails. So the question for me here is: Is behavior B intended? If yes, then libvirt needs to be adapted. If not, it's a systemd bug.

I don't follow here. Can you explain why reseting the PID known state will make this behave like ExitType=cgroup?