Bug: launching nbd-client (and other daemons) in rc.local blocks shutdown in Ubuntu 16.04

[jonsito]

Submission type

• [X ] Bug report
• Request for enhancement (RFE)

NOTE: Do not submit anything other than bug reports or RFEs via the issue tracker!

systemd version the issue has been seen with

Using systemd 229-4ubuntu16 native

NOTE: Do not submit bug reports about anything but the two most recently released systemd versions upstream!

Used distribution

Ubuntu 16.04 fully updated

In case of bug report: Expected behaviour you didn't see

On pressing power button, executing /sbin/poweroff, or click "poweroff" from session ( also with reboot/shutdown ) system must power off

In case of bug report: Unexpected behaviour you saw

System closes session, start poweroff process... and stall.
Sometimes a message is shown:
A stop job is running for /etc/rc.d/rc.local Compatibility (29min 39s / no limit)

In case of bug report: Steps to reproduce the problem

• Edit /etc/rc.local to start nbd-client (or any other daemon that forks/exit )
• Restart computer
• Verify that daemon is running
• Try to shutdown/reboot/poweroff

Additional info:

• The problem is related in the fact that rc.local expect every process started on it should finish and return an exit status.
• As "nbd-client" daemonizes itself, rc.local associated cgroup remains in active/(running) status;
  you can check it by mean of "systemctl status rc.local" . In shutdown, systemd waits for rc.local to finish (as stated "running" ) but has no timeout declared, so waits forever
• if you kill related daemons rc.local service status becomes active/(exited) and shutdown works as expected
• Invoke command by mean of "nohup" or process in background "&" has no effect: PID is anyway associated to rc.local cgroup.
• The only way to start a daemon in rc.local is by mean of "at -f <commandfile_that_includes_daemon> now", as atd runs in a separate cgroup thus rc.local gets no running PIDs in their group

[poettering]

we'll send SIGTERM to all processes rc.local has started during shutdown of that service. Doesn't your service react to that and terminates?

Also, rc-local is a compat kludge, if you have a proper service there's really no point in ever using that.

[poettering]

what does the debian.conf drop-in contain? this appears to be a distro-specific change...

Generally we don#t track issues that aren't in the two most recent systemd versions, which atm is 231 and 232, not 292.

I'd very much prefer if you'd contact your downtsream distro about this first, and let them escalate things if there's really an upstream issue.

[mbiebl]

For completeness sake:

# /lib/systemd/system/rc-local.service.d/debian.conf
[Unit]
# not specified by LSB, but has been behaving that way in Debian under SysV
# init and upstart
After=network-online.target

# Often contains status messages which users expect to see on the console
# during boot
[Service]
StandardOutput=journal+console
StandardError=journal+console

[jonsito]

nbd-client (Network Block Device) is a user space program that handles connection an negotiation with server for using NBD kernel drivers. At Ubuntu 16.04 is not systemd-aware, and in fact traps SIGTERM, and needs to be explicitely invoked to properly handle /dev/nbdX device disconnection

Our environment is a LTSP ( Linux Terminal Server Project) fat (standalone) client, that uses nbd images to deploy hundreds heterogeneus clients. So create and using specific services on server images is not an option. The only way to handle personalization is by mean of rc.local and a (i agree) dirty trick

Anyway, my question is about how to handle nbd-client ( and other non-systemd-aware ) daemons that (for any reason) have to be started in rc.local, and due to their fork+setsid+exit process are retained in rc.local cgroup making rc.local service never exit

[poettering]

nbd-client (Network Block Device) is a user space program that handles connection an negotiation with server for using NBD kernel drivers. At Ubuntu 16.04 is not systemd-aware, and in fact traps SIGTERM, and needs to be explicitely invoked to properly handle /dev/nbdX device disconnection

"traps"? And what does it do on SIGTERM?

On UNIX processes are supposed to exit on SIGTERM. That's what systemd will send any left-over processes in rc.local when shutting down. If the process doesn't react to that, then it is simply broken, and incompatible with UNIX.

My recommendation would be to fix the daemon in question. If that's not possible you can always write a proper service and specify a different command for shutting down the service with ExecStop=. But if you can't do that either, then I cannot really help you: you have a very special and broken daemon there, and it simply doesn't work like a normal UNIX daemon, and hence it's not really compatible with rc.local and normal shutdown behaviour.

Sorry if that's disappointing, but there's nothing we can do about this. Closing.