Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

systemd service fails on Ubuntu 18.04 #138

Closed
alxndr42 opened this issue Jan 3, 2020 · 16 comments · Fixed by #142
Closed

systemd service fails on Ubuntu 18.04 #138

alxndr42 opened this issue Jan 3, 2020 · 16 comments · Fixed by #142
Assignees

Comments

@alxndr42
Copy link

@alxndr42 alxndr42 commented Jan 3, 2020

unbound 1.9.6 fails to load as a systemd service on Ubuntu 18.04.3 LTS, when built from source and using the included contrib/unbound.service file.

unbound was built and installed using the following command line:

./configure --enable-systemd --with-libevent && make && make install && ldconfig

The following error message appears in journalctl:

unbound.service: Failed to set up mount namespacing: No such file or directory
unbound.service: Failed at step NAMESPACE spawning /usr/local/sbin/unbound: No such file or directory
unbound.service: Main process exited, code=exited, status=226/NAMESPACE
unbound.service: Failed with result 'exit-code'.

I narrowed it down to this line in unbound.service:

BindPaths=-/usr/local/etc/unbound/unbound.pid:/usr/local/etc/unbound/usr/local/etc/unbound/unbound.pid

I'm not sure why this causes /usr/local/sbin/unbound to not be found, but when I comment out the line, unbound works as expected. The line itself looks pretty strange, with the duplication of /usr/local/etc/unbound.

@spirillen
Copy link

@spirillen spirillen commented Jan 3, 2020

Hi @7adietri could you try this build line?

https://gitlab.com/rpz-zones/toolbox/issues/18 It is working for me on Ubuntu18/19 from at least version 1.9.4 to 1.9.6

@alxndr42
Copy link
Author

@alxndr42 alxndr42 commented Jan 3, 2020

Hi @spirillen, are you referring to this command line?
https://gitlab.com/rpz-zones/toolbox/issues/18#configure-options

Since I don't want to install into /usr, I just tried a build using --with-pidfile=/usr/local/etc/unbound/unbound.pid, but the resulting unbound.service file still contains the weird duplication of the path.

I wonder why this line makes /usr/local/sbin/unbound "disappear" for the process, that seems completely random.

@spirillen
Copy link

@spirillen spirillen commented Jan 3, 2020

Hi @7adietri Yes that's the line

If I recall it right, then I chosen to install into the /usr because of the AppAmor /etc/apparmor.d/usr.sbin.unbound as that would make a lot of noice.

PS I'm not a unbound guru, just doing my best to help you up and running 😄

@Maryse47
Copy link
Contributor

@Maryse47 Maryse47 commented Jan 4, 2020

Could you paste whole .service file + unbound config? Storing pidfile in etc is bit unusual, could you try --with-pidfile=/var/run/unbound.pid?

@alxndr42
Copy link
Author

@alxndr42 alxndr42 commented Jan 4, 2020

This is the complete unbound.service file generated by make:

[Unit]
Description=Validating, recursive, and caching DNS resolver
Documentation=man:unbound(8)
After=network.target
Before=network-online.target nss-lookup.target
Wants=nss-lookup.target

[Install]
WantedBy=multi-user.target

[Service]
ExecReload=+/bin/kill -HUP $MAINPID
ExecStart=/usr/local/sbin/unbound -d
NotifyAccess=main
Type=notify
CapabilityBoundingSet=CAP_IPC_LOCK CAP_NET_BIND_SERVICE CAP_SETGID CAP_SETUID CAP_SYS_CHROOT CAP_SYS_RESOURCE CAP_NET_RAW
MemoryDenyWriteExecute=true
NoNewPrivileges=true
PrivateDevices=true
PrivateTmp=true
ProtectHome=true
ProtectControlGroups=true
ProtectKernelModules=true
ProtectSystem=strict
ReadWritePaths=/run /usr/local/etc/unbound /usr/local/etc/unbound
TemporaryFileSystem=/usr/local/etc/unbound/dev:ro
TemporaryFileSystem=/usr/local/etc/unbound/run:ro
BindReadOnlyPaths=-/run/systemd/notify:/usr/local/etc/unbound/run/systemd/notify
BindPaths=-/usr/local/etc/unbound/unbound.pid:/usr/local/etc/unbound/usr/local/etc/unbound/unbound.pid
BindReadOnlyPaths=-/dev/urandom:/usr/local/etc/unbound/dev/urandom
BindPaths=-/dev/log:/usr/local/etc/unbound/dev/log
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
RestrictRealtime=true
SystemCallArchitectures=native
SystemCallFilter=~@clock @cpu-emulation @debug @keyring @module mount @obsolete @resources
RestrictNamespaces=yes
LockPersonality=yes
RestrictSUIDSGID=yes

The service fails to start unless I remove the line starting with BindPaths=-/usr/local/etc/unbound.

The only relevant setting from the config file is probably chroot: "/usr/local/etc/unbound".

I don't want to install manually built software into the same locations as packaged software, so the ./configure default of using /usr/local is fine with me. As I mentioned above, I tried --with-pidfile=/usr/local/etc/unbound/unbound.pid, but the generated unbound.service file doesn't change. It's still not clear to me why this has an effect on the visibility of /usr/local/sbin/unbound.

@Maryse47
Copy link
Contributor

@Maryse47 Maryse47 commented Jan 4, 2020

The unbound.pid isn't something created during install but a temporary file created during runtime. It won't conflict with packaged software and all .pid files are usually created either in /run or /var/run which are mounted as tmpfs. That's why setting --with-pidfile=/run/unbound.pid should be safe thing to do for you.

You may also clean leftovers from /usr/local/etc/unbound

@Maryse47
Copy link
Contributor

@Maryse47 Maryse47 commented Jan 4, 2020

it would be also nice to rule out apparmor issues: journalctl | grep DENIED

@alxndr42
Copy link
Author

@alxndr42 alxndr42 commented Jan 4, 2020

The PID file is created in /usr/local/etc/unbound and seems to be working fine, the actual error message is oddly enough about /usr/local/sbin/unbound.

There's no occurrence of DENIED in the journalctl output, just what I posted above.

@Maryse47
Copy link
Contributor

@Maryse47 Maryse47 commented Jan 4, 2020

I understand but as you said this line is causing issues for you. Maybe something gets confused when it tries bind-mount file from the same dir (which I didn't expect to happen). I'll appreciate if you could just test what happens if you change it.

@alxndr42
Copy link
Author

@alxndr42 alxndr42 commented Jan 5, 2020

When I build with --with-pidfile=/run/unbound.pid, the PID file is created in /run, but the contents of unbound.service and the runtime behaviour don't change. I still have to comment out the line to make unbound start. I'll leave it commented out for now and try to investigate further.

@Maryse47
Copy link
Contributor

@Maryse47 Maryse47 commented Jan 5, 2020

Hm, this is impossible to happen. I confirmed locally that --with-pidfile affects UNBOUND_PIDFILE macro used in service. I guess your build reused some artifacts from previous one. Please try cleaning all build artifacts, systemd services and /usr/local/etc/unbound contencts and try again.

@alxndr42
Copy link
Author

@alxndr42 alxndr42 commented Jan 5, 2020

Since make clean doesn't remove contrib/unbound.service, I deleted it manually and then built again. This time the line changed to:

BindPaths=-/run/unbound.pid:/usr/local/etc/unbound/run/unbound.pid

But the behaviour is still the same, the service fails because /usr/local/sbin/unbound is "missing", unless I comment out the line. I'll try to do some more debugging when I have the time.

@Maryse47
Copy link
Contributor

@Maryse47 Maryse47 commented Jan 6, 2020

Did you removed stale /usr/local/etc/unbound/ content (everything other than your config)?

@ralphdolmans ralphdolmans self-assigned this Jan 6, 2020
@alxndr42
Copy link
Author

@alxndr42 alxndr42 commented Jan 7, 2020

Yes, /usr/local/etc/unbound contained only the config file and keys.

@ralphdolmans
Copy link
Contributor

@ralphdolmans ralphdolmans commented Jan 8, 2020

Hi,

But the behaviour is still the same, the service fails because /usr/local/sbin/unbound is "missing", unless I comment out the line. I'll try to do some more debugging when I have the time.

I don't think you get the error because sbin/unbound is missing. You get an error that something is missing while setting up the name space, which happens before spawning sbin/unbound.

The missing part is the source part of the BindPaths property, which is the pid file. This file does indeed not exist when Unbound is not running. The systemd service file states that it can be ignored if it does not exist (the "-" prefix), but I don't think that is supported in the systemd version shipped with Ubuntu 18.04.

I am wondering if this BindPaths line is needed at all. The pidfile is created before dropping privileges. And if I understand the systemd docs correctly it will be ignored most of the time (when the pid does not exist) anyway.

@Maryse47
Copy link
Contributor

@Maryse47 Maryse47 commented Jan 8, 2020

Ok, I created #142

ralphdolmans added a commit that referenced this issue Jan 8, 2020
jedisct1 added a commit to jedisct1/unbound that referenced this issue Jan 27, 2020
* nlnet/master: (28 commits)
  - Add changelog entry for PR#148.
  - Add changelog entry for RP#154 - autoconf after PR#154
  - Fix NLnetLabs#153: Disable validation for DSA algorithms.  RFC 8624   compliance.
  Changelog note for PR#155. - Merge PR#155 from Rober Edmonds: contrib/libunbound.pc.in: Fixes   to Libs/Requires for crypto library dependencies.
  contrib/libunbound.pc.in: Embed the correct crypto dependencies
  contrib/libunbound.pc.in: Only specify -lunbound for Libs
  Allow use of libbsd functions with configure option --with-libbsd
  Changelog and contrib/README note for PR#150. - Merge PR#150 from Frzk: Systemd unit without chroot.  It add   contrib/unbound_nochroot.service.in, a systemd file for use with   chroot: "", see comments in the file, it uses systemd protections   instead.
  Patch configure.ac file to take the new contrib/unbound_nochroot.service unit file in consideration.
  Added a new unit file to run unbound with systemd and without chroot.
  Update unbound_munin_
  - Fix auth zone support for NSEC3 records without salt.
  - Fix for memory leak when edns subnet config options are read when   compiled without edns subnet support.
  - Fix crash after reload where a stats lookup could reference old key   cache and neg cache structures.
  - Removed the dnscrypt_queries and dnscrypt_queries_chacha tests,   because dnscrypt-proxy (2.0.36) does not support the test setup   any more, and also the config file format does not seem to have   the appropriate keys to recreate that setup.
  - Fix unreachable code in ssl set options code.
  - Fix the relationship between serve-expired and prefetch options,   patch from Saksham Manchanda from Secure64.
  - Add changelog entry for fix NLnetLabs#138 (stop binding pidfile inside chroot dir in   systemd service file).
  unbound.service.in: stop binding pidfile inside chroot dir
  - And update for more spare space.
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants