dhcpcd occasionally takes 20 seconds to stop, slowing reboot

[archenemies]

systemd version the issue has been seen with

245

Used distribution

Arch

Expected behaviour you didn't see

It should be easy to stop dhcpcd

Unexpected behaviour you saw

Running systemctl stop dhcpcd@en0.service sometimes takes 20 seconds, as if it is timing out. This never happens with sudo dhcpcd -x en0. There are no error messages. The timeout makes reboot very slow.

$ time sudo systemctl stop dhcpcd@en0.service
sudo systemctl stop dhcpcd@en0.service  0.02s user 0.01s system 38% cpu 0.083 total
$ time sudo systemctl start dhcpcd@en0.service
sudo systemctl start dhcpcd@en0.service  0.01s user 0.02s system 46% cpu 0.070 total
$ time sudo systemctl stop dhcpcd@en0.service
sudo systemctl stop dhcpcd@en0.service  0.02s user 0.01s system 0% cpu 20.144 total

Here are some journal entries:

Jun 01 10:23:23 ptolemy systemd[1]: dhcpcd@en0.service: State 'stop-sigterm' timed out. Killing.
Jun 01 10:23:23 ptolemy systemd[1]: dhcpcd@en0.service: Killing process 4919 (dhcpcd) with signal SIGKILL.
Jun 01 10:23:23 ptolemy systemd[1]: dhcpcd@en0.service: Main process exited, code=killed, status=9/KILL
Jun 01 10:23:23 ptolemy systemd[1]: dhcpcd@en0.service: Failed with result 'timeout'.
Jun 01 10:23:23 ptolemy systemd[1]: Stopped dhcpcd on en0.

Steps to reproduce the problem

[poettering]

dhcpcd does not react to SIGTERM apparently. please report to your dhcpcd downstream.

[archenemies]

I don't fully understand the problem yet. In /etc/systemd/system/dhcpcd.service I find:

ExecStop=/usr/bin/dhcpcd -x

This command works reliably when I run it from a terminal.

Also, sometimes the following command executes quickly as well, as I showed above:

systemctl stop dhcpcd@en0.service

Are you suggesting that in addition to "ExecStop=/usr/bin/dhcpcd -x", Systemd is sending a SIGTERM on its own initiative? I checked that dhcpcd doesn't exit on SIGTERM, but I'm not sure how or why that became a standard requirement for daemons.

Here is the unit file in case it helps:

$ cat /etc/systemd/system/dhcpcd.service
[Unit]
Description=dhcpcd on all interfaces
Wants=network.target
Before=network.target

[Service]
Type=forking
PIDFile=/run/dhcpcd/pid
ExecStart=/usr/bin/dhcpcd -q -b
ExecStop=/usr/bin/dhcpcd -x

[Install]
WantedBy=multi-user.target

[floppym]

dhcpcd.service and dhcpcd@.service are separate units. You are looking at the wrong one.

This is not a systemd bug. Please seek support from the Arch people.

[poettering]

@archenemies systemd will send SIGTERM to the remaining processes when there are any left-over processes after the ExecStop= command completed. It appers dhcpd doesn't react to neither "dhcpcd -x" nor to SIGTERM. Which is something you need to report to the dhcpcd package.

(Maybe you have some hook scripts defined that dhcpcd waits on and block shutdown?)

Either way, this is unlikely to be a systemd problem. please report to your downstream distro, in the dhcpcd package.

[archenemies]

I understood your repeated request that I should report it upstream or downstream, I'm just trying to understand what I should report.

According to my experiments, there is only a problem when dhcpcd -x en0 is invoked via Systemd, if I run it on the command line then the dhcpcd always exits immediately.

I think the only customization I have is this:

$ cat /etc/systemd/system/dhcpcd@.service.d/no-wait.conf
[Service]
ExecStart=
ExecStart=/usr/bin/dhcpcd -b -q %I

I will send an email to the Dhcpcd mailing list.

[rsmarples]

This seems to be an Arch + systemd only issue.
Someone kindly tested the same dhcpcd setup on Gentoo where it worked without issue.
dhcpcd reacts just fine to SIGTERM as any well behaved process should across all versions.

In a nutshell, for dhcpcd-9.0 there is a chroot setup script which mount --bind /dev, /dev/rlog, /run, /proc and /sys in the chroot so that the chrooted process could access the necessary.
For dhcpcd-9.1, this has been changed so that these are accessed from the chroot via IPC so no mounting is necessary.

I have no explanation as to why this is an issue at all - AFAIK it only affects systemd on Arch.
systemd on Gentoo worked fine according to @floppym . It also worked fine using other init systems.

[archenemies]

@rsmarples Thank you Roy. After upgrading dhcpcd I verified that the problem is gone.

[poettering]

Ah, if the problem is gone, let's close it here then.