systemd service fails on Ubuntu 18.04

[alxndr42]

unbound 1.9.6 fails to load as a systemd service on Ubuntu 18.04.3 LTS, when built from source and using the included contrib/unbound.service file.

unbound was built and installed using the following command line:

./configure --enable-systemd --with-libevent && make && make install && ldconfig

The following error message appears in journalctl:

unbound.service: Failed to set up mount namespacing: No such file or directory
unbound.service: Failed at step NAMESPACE spawning /usr/local/sbin/unbound: No such file or directory
unbound.service: Main process exited, code=exited, status=226/NAMESPACE
unbound.service: Failed with result 'exit-code'.

I narrowed it down to this line in unbound.service:

BindPaths=-/usr/local/etc/unbound/unbound.pid:/usr/local/etc/unbound/usr/local/etc/unbound/unbound.pid

I'm not sure why this causes /usr/local/sbin/unbound to not be found, but when I comment out the line, unbound works as expected. The line itself looks pretty strange, with the duplication of /usr/local/etc/unbound.

[spirillen]

Hi @7adietri could you try this build line?

https://gitlab.com/rpz-zones/toolbox/issues/18 It is working for me on Ubuntu18/19 from at least version 1.9.4 to 1.9.6

[alxndr42]

Hi @spirillen, are you referring to this command line?
https://gitlab.com/rpz-zones/toolbox/issues/18#configure-options

Since I don't want to install into /usr, I just tried a build using --with-pidfile=/usr/local/etc/unbound/unbound.pid, but the resulting unbound.service file still contains the weird duplication of the path.

I wonder why this line makes /usr/local/sbin/unbound "disappear" for the process, that seems completely random.

[spirillen]

Hi @7adietri Yes that's the line

If I recall it right, then I chosen to install into the /usr because of the AppAmor /etc/apparmor.d/usr.sbin.unbound as that would make a lot of noice.

PS I'm not a unbound guru, just doing my best to help you up and running 😄

[Maryse47]

Could you paste whole .service file + unbound config? Storing pidfile in etc is bit unusual, could you try --with-pidfile=/var/run/unbound.pid?

[alxndr42]

This is the complete unbound.service file generated by make:

[Unit]
Description=Validating, recursive, and caching DNS resolver
Documentation=man:unbound(8)
After=network.target
Before=network-online.target nss-lookup.target
Wants=nss-lookup.target

[Install]
WantedBy=multi-user.target

[Service]
ExecReload=+/bin/kill -HUP $MAINPID
ExecStart=/usr/local/sbin/unbound -d
NotifyAccess=main
Type=notify
CapabilityBoundingSet=CAP_IPC_LOCK CAP_NET_BIND_SERVICE CAP_SETGID CAP_SETUID CAP_SYS_CHROOT CAP_SYS_RESOURCE CAP_NET_RAW
MemoryDenyWriteExecute=true
NoNewPrivileges=true
PrivateDevices=true
PrivateTmp=true
ProtectHome=true
ProtectControlGroups=true
ProtectKernelModules=true
ProtectSystem=strict
ReadWritePaths=/run /usr/local/etc/unbound /usr/local/etc/unbound
TemporaryFileSystem=/usr/local/etc/unbound/dev:ro
TemporaryFileSystem=/usr/local/etc/unbound/run:ro
BindReadOnlyPaths=-/run/systemd/notify:/usr/local/etc/unbound/run/systemd/notify
BindPaths=-/usr/local/etc/unbound/unbound.pid:/usr/local/etc/unbound/usr/local/etc/unbound/unbound.pid
BindReadOnlyPaths=-/dev/urandom:/usr/local/etc/unbound/dev/urandom
BindPaths=-/dev/log:/usr/local/etc/unbound/dev/log
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
RestrictRealtime=true
SystemCallArchitectures=native
SystemCallFilter=~@clock @cpu-emulation @debug @keyring @module mount @obsolete @resources
RestrictNamespaces=yes
LockPersonality=yes
RestrictSUIDSGID=yes

The service fails to start unless I remove the line starting with BindPaths=-/usr/local/etc/unbound.

The only relevant setting from the config file is probably chroot: "/usr/local/etc/unbound".

I don't want to install manually built software into the same locations as packaged software, so the ./configure default of using /usr/local is fine with me. As I mentioned above, I tried --with-pidfile=/usr/local/etc/unbound/unbound.pid, but the generated unbound.service file doesn't change. It's still not clear to me why this has an effect on the visibility of /usr/local/sbin/unbound.

[Maryse47]

The unbound.pid isn't something created during install but a temporary file created during runtime. It won't conflict with packaged software and all .pid files are usually created either in /run or /var/run which are mounted as tmpfs. That's why setting --with-pidfile=/run/unbound.pid should be safe thing to do for you.

You may also clean leftovers from /usr/local/etc/unbound

[Maryse47]

it would be also nice to rule out apparmor issues: journalctl | grep DENIED

[alxndr42]

The PID file is created in /usr/local/etc/unbound and seems to be working fine, the actual error message is oddly enough about /usr/local/sbin/unbound.

There's no occurrence of DENIED in the journalctl output, just what I posted above.

[Maryse47]

I understand but as you said this line is causing issues for you. Maybe something gets confused when it tries bind-mount file from the same dir (which I didn't expect to happen). I'll appreciate if you could just test what happens if you change it.

[alxndr42]

When I build with --with-pidfile=/run/unbound.pid, the PID file is created in /run, but the contents of unbound.service and the runtime behaviour don't change. I still have to comment out the line to make unbound start. I'll leave it commented out for now and try to investigate further.

[Maryse47]

Hm, this is impossible to happen. I confirmed locally that --with-pidfile affects UNBOUND_PIDFILE macro used in service. I guess your build reused some artifacts from previous one. Please try cleaning all build artifacts, systemd services and /usr/local/etc/unbound contencts and try again.

[alxndr42]

Since make clean doesn't remove contrib/unbound.service, I deleted it manually and then built again. This time the line changed to:

BindPaths=-/run/unbound.pid:/usr/local/etc/unbound/run/unbound.pid

But the behaviour is still the same, the service fails because /usr/local/sbin/unbound is "missing", unless I comment out the line. I'll try to do some more debugging when I have the time.

[Maryse47]

Did you removed stale /usr/local/etc/unbound/ content (everything other than your config)?

[alxndr42]

Yes, /usr/local/etc/unbound contained only the config file and keys.