Service Failed with 'timeout' #56
Comments
|
I see that you are using debian, is this a packaged unbound or you build your own from source? |
|
Yes - it's straight from the repo: |
|
I am not able to reproduce the problem with a fresh installation of debian buster. This included installing the OS and apt installing unbound. I cannot reproduce the problem with any combination of forwarding servers and DNS over TLS. Have you installed unbound in another way, or do you use a more complicated configuration? By the way, the ripe name servers that you use were only set up for that conference if I recall correctly as they are not responding now (this does not cause any problems here with regards to systemd except that queries are not answered). |
|
Thanks for looking into this. That was a sample of two to demonstrate configuration - there are many other DoT configured in the "." forward zone, and resolution is 100% operational during the time unbound is actually up. Resolution is not the issue, the service start / timeout is the problem. I deployed unbound using standard repo tools e.g. Have you tried using Debian Buster on an arm64 architecture device? Do you have any recommendations for me to try? |
|
I wondered if there might have been an issue resolved during the transition of Buster from testing to stable, as unbound was originally installed when Buster was still in testing. So just did an |
|
I could also try an arm64 installation but I am not expecting to find something different. Before I go down that road it could be useful to share your configuration (minus any sensitive settings). |
|
Sure - configs below. I've variously tried removing specific config files to see if any of them contain configurations which would do this, but have not seen the problem resolve itself in these tests. unbound.conf unbound.conf.d/base.conf unbound.conf.d/caching.conf unbound.conf.d/tls.conf unbound.conf.d/root-auto-trust-anchor-file.conf unbound.conf.d/qname-min.conf |
|
I was finally able to reproduce the issue. The offending option was I would advise to remove the Can you verify that without chroot it works as expected? I will consider opening an issue to inform the user if such configuration is encountered and immediately terminate to avoid cryptic behavior. |
|
Sorry for taking so long to reply, was away for a few days and could not find a secure network. I've just tried it without chroot and you're right - it works without issue. The Confining DNS via chroot is useful from a security perspective due to natural exposure of such a system. I've used bind mounts to deal with log and cert folders within the confines of the chroot - is there something similar I should do for this socket? |
|
I see two options. Both are not ideal but they work around the issue:
|
|
Hi - option #1 is the most viable to ensure upgrades are not affected / do not affect the operation of the service. I tried binding I couldn't find any references in man for unbound or systemd service unit files - did I miss something? Essentially this is a defect with the chroot option which needs resolution by:
|
|
Hi, now it is my time to apologize for the late response :) When you tried binding the folder did you end up with |
|
Bind-mounting systemd socket should be fixed by ff8fd0b . I don't know if debian will use upstream service file though. |
No problem at all :) Did some digging and I'd made a mistake: I hadn't created the path structure correctly for the bind mount (looks like I'd forgotten the systemd subfolder). Corrected so that the mount destination was /etc/unbound/run/systemd and it appears to be working. Appears to be running within chroot with the workaround you've described. Thanks for your help getting this operational. |
Thanks for that - looks like 1.9.3-1 recently got pulled into Debian testing, so will keep an eye out for versions with your fix. A side note: I didn't appear to need to bind |
I think at some point unbound was teached to read from /dev/urandom before chrooting. Some docs still point to bind-mounting /dev/urandom and I added it just in case to be safe. I don't know if it's useful but it shouldn't do any harm. |
|
On linux kernels 3.17+ it will also call |
|
In Debian 10/Buster the standard location for the chroot is
When unbound is started, the configurations files are copied from All this is not even mentioned by the Debian documentation and also not the correct location of the chroot ( Thanks. |
|
I'm seeing serious issues with unbound where the service simply restarts itself intermittently. I say intermittently but I'm beginning to think it's related to the service failed timeout + the restart delay.
When I attempt to start the service normally via
systemctl start unboundorsystemctl restart unboundthe prompt hangs and waits. Eventually there's an error message along the lines of:Job for unbound.service failed because a timeout was exceeded. See "systemctl status unbound.service" and "journalctl -xe" for details.In the unbound log I can see that it's reached a status of
info: start of service (unbound 1.9.0).within milliseconds of the service start though.tls-upstream is set to yes, and looking at the more verbose outputs requests are getting resolved. However when the service restarts the cache is dropped, which is the one of the main reasons for using unbound on this LAN (as well as part of a wider security & privacy design).
I'm using DNS-over-TLS upstreams only, and are configured as per this example:
I know this generally works because if I start the service directly (stopping the systemd service) via
/usr/sbin/unbound -dvv, I see it working as expected. No restarts or drop-outs.It's almost like the systemd service thinks the startup hasn't completed, and assumes timeout - and forces the restart based on the unbound.service
Restart=on-failedsetting. If I remove theRestart=...line from the service I see the following in thesystemctl status:The pre-reqs for chroot and trust anchors appear to operate as expected, it's the actual unbound service which seems to timeout as a systemd service.
The server in question is a new build to replace an older server, running another Debian derivative with unbound 1.6.0, and that uses Stubby and DnsCrypt as its forwarders. I've tried swapping out the DoT configuration for a straight udp forwarder with known-good DNS (Cloudflare in the test), and that exhibits the same symptoms.
Version in use: 1.9.0.2
OS: Debian Buster (stable) arm64
I enabled a higher verbosity and captured a part of the log for reference - it shows the brief time unbound was actually up through to the timeout 'happening' and the service dropping. Just after the
systemctl start unboundI hit it with adig google.com +dnssec:The text was updated successfully, but these errors were encountered: