systemd service fails on Ubuntu 18.04

[alxndr42]

unbound 1.9.6 fails to load as a systemd service on Ubuntu 18.04.3 LTS, when built from source and using the included contrib/unbound.service file.

unbound was built and installed using the following command line:

./configure --enable-systemd --with-libevent && make && make install && ldconfig

The following error message appears in journalctl:

unbound.service: Failed to set up mount namespacing: No such file or directory
unbound.service: Failed at step NAMESPACE spawning /usr/local/sbin/unbound: No such file or directory
unbound.service: Main process exited, code=exited, status=226/NAMESPACE
unbound.service: Failed with result 'exit-code'.

I narrowed it down to this line in unbound.service:

BindPaths=-/usr/local/etc/unbound/unbound.pid:/usr/local/etc/unbound/usr/local/etc/unbound/unbound.pid

I'm not sure why this causes /usr/local/sbin/unbound to not be found, but when I comment out the line, unbound works as expected. The line itself looks pretty strange, with the duplication of /usr/local/etc/unbound.

[spirillen]

Hi @7adietri could you try this build line?

https://gitlab.com/rpz-zones/toolbox/issues/18 It is working for me on Ubuntu18/19 from at least version 1.9.4 to 1.9.6

[alxndr42]

Hi @spirillen, are you referring to this command line?
https://gitlab.com/rpz-zones/toolbox/issues/18#configure-options

Since I don't want to install into /usr, I just tried a build using --with-pidfile=/usr/local/etc/unbound/unbound.pid, but the resulting unbound.service file still contains the weird duplication of the path.

I wonder why this line makes /usr/local/sbin/unbound "disappear" for the process, that seems completely random.

[spirillen]

Hi @7adietri Yes that's the line

If I recall it right, then I chosen to install into the /usr because of the AppAmor /etc/apparmor.d/usr.sbin.unbound as that would make a lot of noice.

PS I'm not a unbound guru, just doing my best to help you up and running 😄

[Maryse47]

Could you paste whole .service file + unbound config? Storing pidfile in etc is bit unusual, could you try --with-pidfile=/var/run/unbound.pid?

[alxndr42]

This is the complete unbound.service file generated by make:

[Unit]
Description=Validating, recursive, and caching DNS resolver
Documentation=man:unbound(8)
After=network.target
Before=network-online.target nss-lookup.target
Wants=nss-lookup.target

[Install]
WantedBy=multi-user.target

[Service]
ExecReload=+/bin/kill -HUP $MAINPID
ExecStart=/usr/local/sbin/unbound -d
NotifyAccess=main
Type=notify
CapabilityBoundingSet=CAP_IPC_LOCK CAP_NET_BIND_SERVICE CAP_SETGID CAP_SETUID CAP_SYS_CHROOT CAP_SYS_RESOURCE CAP_NET_RAW
MemoryDenyWriteExecute=true
NoNewPrivileges=true
PrivateDevices=true
PrivateTmp=true
ProtectHome=true
ProtectControlGroups=true
ProtectKernelModules=true
ProtectSystem=strict
ReadWritePaths=/run /usr/local/etc/unbound /usr/local/etc/unbound
TemporaryFileSystem=/usr/local/etc/unbound/dev:ro
TemporaryFileSystem=/usr/local/etc/unbound/run:ro
BindReadOnlyPaths=-/run/systemd/notify:/usr/local/etc/unbound/run/systemd/notify
BindPaths=-/usr/local/etc/unbound/unbound.pid:/usr/local/etc/unbound/usr/local/etc/unbound/unbound.pid
BindReadOnlyPaths=-/dev/urandom:/usr/local/etc/unbound/dev/urandom
BindPaths=-/dev/log:/usr/local/etc/unbound/dev/log
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
RestrictRealtime=true
SystemCallArchitectures=native
SystemCallFilter=~@clock @cpu-emulation @debug @keyring @module mount @obsolete @resources
RestrictNamespaces=yes
LockPersonality=yes
RestrictSUIDSGID=yes

The service fails to start unless I remove the line starting with BindPaths=-/usr/local/etc/unbound.

The only relevant setting from the config file is probably chroot: "/usr/local/etc/unbound".

I don't want to install manually built software into the same locations as packaged software, so the ./configure default of using /usr/local is fine with me. As I mentioned above, I tried --with-pidfile=/usr/local/etc/unbound/unbound.pid, but the generated unbound.service file doesn't change. It's still not clear to me why this has an effect on the visibility of /usr/local/sbin/unbound.

[Maryse47]

The unbound.pid isn't something created during install but a temporary file created during runtime. It won't conflict with packaged software and all .pid files are usually created either in /run or /var/run which are mounted as tmpfs. That's why setting --with-pidfile=/run/unbound.pid should be safe thing to do for you.

You may also clean leftovers from /usr/local/etc/unbound

[Maryse47]

it would be also nice to rule out apparmor issues: journalctl | grep DENIED

[alxndr42]

The PID file is created in /usr/local/etc/unbound and seems to be working fine, the actual error message is oddly enough about /usr/local/sbin/unbound.

There's no occurrence of DENIED in the journalctl output, just what I posted above.

[Maryse47]

I understand but as you said this line is causing issues for you. Maybe something gets confused when it tries bind-mount file from the same dir (which I didn't expect to happen). I'll appreciate if you could just test what happens if you change it.

[alxndr42]

When I build with --with-pidfile=/run/unbound.pid, the PID file is created in /run, but the contents of unbound.service and the runtime behaviour don't change. I still have to comment out the line to make unbound start. I'll leave it commented out for now and try to investigate further.

[Maryse47]

Hm, this is impossible to happen. I confirmed locally that --with-pidfile affects UNBOUND_PIDFILE macro used in service. I guess your build reused some artifacts from previous one. Please try cleaning all build artifacts, systemd services and /usr/local/etc/unbound contencts and try again.

[alxndr42]

Since make clean doesn't remove contrib/unbound.service, I deleted it manually and then built again. This time the line changed to:

BindPaths=-/run/unbound.pid:/usr/local/etc/unbound/run/unbound.pid

But the behaviour is still the same, the service fails because /usr/local/sbin/unbound is "missing", unless I comment out the line. I'll try to do some more debugging when I have the time.

[Maryse47]

Did you removed stale /usr/local/etc/unbound/ content (everything other than your config)?

[alxndr42]

Yes, /usr/local/etc/unbound contained only the config file and keys.

[ralphdolmans]

Hi,

But the behaviour is still the same, the service fails because /usr/local/sbin/unbound is "missing", unless I comment out the line. I'll try to do some more debugging when I have the time.

I don't think you get the error because sbin/unbound is missing. You get an error that something is missing while setting up the name space, which happens before spawning sbin/unbound.

The missing part is the source part of the BindPaths property, which is the pid file. This file does indeed not exist when Unbound is not running. The systemd service file states that it can be ignored if it does not exist (the "-" prefix), but I don't think that is supported in the systemd version shipped with Ubuntu 18.04.

I am wondering if this BindPaths line is needed at all. The pidfile is created before dropping privileges. And if I understand the systemd docs correctly it will be ignored most of the time (when the pid does not exist) anyway.

[Maryse47]

Ok, I created #142