You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
fatal error: could not open autotrust file for writing, /var/lib/unbound/root.key.2602864-1-560c3a7457d0: Permission denied #447
Unbound periodically writes to the root.key file, to process the updates for RFC5011 updates. That is what you have configured.
The error is that unbound wants to create a temporary file to write the new contents of the /var/lib/unbound/root.key into. After that unbound is going to remove the old root.key file and rename this file to the root.key file. But unbound gets permission denied when it attempts to write there. Perhaps the directory has no write permissions (for the unbound user), or you have selinux or apparmor or freebsd capabilities configured to deny write access from unbound to this directory?
Not sure if this is the long term fix but I ran chown -R unbound:unbound /var/lib/unbound and restarted unbound but I get these error: read (in tcp s): Connection refused as mentioned in issue 360.
Mar 19 12:15:52 ourserver unbound[2660803]: [2660803:3] error: read (in tcp s): Connection refused for 199.180.182.53 port 53
Mar 19 12:15:52 ourserver unbound[2660803]: [2660803:3] error: read (in tcp s): Connection refused for 199.180.182.53 port 53
Mar 19 12:15:52 ourserver unbound[2660803]: [2660803:3] error: read (in tcp s): Connection refused for 193.0.9.1 port 53
Mar 19 12:15:52 ourserver unbound[2660803]: [2660803:3] error: read (in tcp s): Connection refused for 193.0.9.1 port 53
Mar 19 12:15:52 ourserver unbound[2660803]: [2660803:3] error: read (in tcp s): Connection refused for 196.216.169.10 port 53
Mar 19 12:15:52 ourserver unbound[2660803]: [2660803:3] error: read (in tcp s): Connection refused for 196.216.169.10 port 53
Mar 19 12:15:52 ourserver unbound[2660803]: [2660803:3] error: read (in tcp s): Connection refused for 193.0.9.1 port 53
Mar 19 12:15:52 ourserver unbound[2660803]: [2660803:3] error: read (in tcp s): Connection refused for 193.0.9.1 port 53
Mar 19 12:15:52 ourserver unbound[2660803]: [2660803:3] error: read (in tcp s): Connection refused for 200.10.60.53 port 53
Mar 19 12:15:52 ourserver unbound[2660803]: [2660803:3] error: read (in tcp s): Connection refused for 200.10.60.53 port 53
TCP Fast Open is disabled. How can I debug this?
dig +tcp @193.0.9.1
;; Connection to 193.0.9.1#53(193.0.9.1) for . failed: connection refused.
The chown of the directory worked, and that fixed your permission denied failure that you had.
The connection refused, is a spammy message that says unbound cannot make tcp connections. I have made it less visible, by only making it print when verbosity is high. This is committed to the repository.
It seems you cannot make TCP connections, and dig also cannot make tcp connections to those IP addresses from the commandline. I also do not know why the connections are refused by your upstream. Perhaps your firewall is refusing to allow tcp connections there, or another firewall is refusing tcp connections. This is a separate issue from your permission denied issue.
Once I removed the -d option from /usr/lib/systemd/system/unbound.service in ExecStart the TCP timeout issues also disappeared from the logs. Perhaps this was already done upstream?
Anyways, as for the original issue, yes it's fixed. I haven't tried a reboot but I'm assuming it should be persistent. But I don't see this documented anywhere to change the permission in the/var/lib/unbounddirectory. Should that be part of the installation process?
Not sure how the removal of the -d option fixes your issue, perhaps what happened was you reloaded your new config file in the process with fixed options in it? Or it fixed it somehow, the unbound.service file is part of the unbound rpm that you are using.
Also the var/lib/unbound directory is created by the rpm and also the configuration that unbound uses that directory to store files is part of the rpm configure script (or default configuration shipped with the package). But we do not do the package scripts, that is handled by the distro that you are using. A look at the unbound.spec file shows that it performs a chown for the user unbound on install %dir %attr(0755,unbound,unbound) %{_sharedstatedir}/%{name}, so I do not know what is wrong with that.
Not sure how the removal of the -d option fixes your issue, perhaps what happened was you reloaded your new config file in the process with fixed options in it? Or it fixed it somehow, the unbound.service file is part of the unbound rpm that you are using.
I can confirm that on a second server running unbound, that when I removed the -d option from the /usr/lib/systemd/system/unbound.service all of those error: read (in tcp s): Connection refused errors disappeared.
The -d option, documented with -d do not fork into the background., makes unbound not fork into a new process. So with -d it stays attached to the console on which it was started. This is what systemd wants because systemd is monitoring that console.
Systemd may also have set other permissions on that console, did it? Would that have caused the different results for tcp connections for the program? Without the -d option unbound forks into a process, but I do not really see what difference that could have made to the tcp connections, so that it gives no connection refused.
OK a little embarrassing here. The -d option is definitely needed. interface-automatic was set to the default, no.
# NOTE: Enable this option when specifying interface 0.0.0.0 or ::0
# NOTE: Disabled per Fedora policy not to listen to * on default install
# NOTE: If deploying on non-default port, eg 80/443, this needs to be disabled
interface-automatic: yes
I'm still getting the error: read (in tcp s): Connection refused for errors but at least unbound is working.
dig +tcp @193.0.9.1
;; Connection to 193.0.9.1#53(193.0.9.1) for . failed: connection refused.
Perhaps it's not TCP but DNSSEC that is expected? See logs.
Mar 31 09:42:36 unbound[3903467:1] info: sending query: fedoraproject.org. A IN
Mar 31 09:42:36 unbound[3903467:1] debug: sending to target: <.> 2001:500:a8::e#53
Mar 31 09:42:36 unbound[3903467:1] debug: dnssec status: expected
Mar 31 09:42:36 unbound[3903467:1] debug: EDNS lookup known=0 vs=0
Mar 31 09:42:36 unbound[3903467:1] debug: serviced query UDP timeout=376 msec
Mar 31 09:42:36 unbound[3903467:1] debug: inserted new pending reply id=ed4f
Mar 31 09:42:36 unbound[3903467:1] debug: opened UDP if=0 port=55718
Mar 31 09:42:36 unbound[3903467:1] error: udp connect failed: Network is unreachable for 2001:500:a8::e port 53 (len 28)
Mar 31 09:42:36 unbound[3903467:1] info: error sending query to auth server ip6 2001:500:a8::e port 53 (len 28)
Mar 31 09:42:36 unbound[3903467:1] debug: iter_handle processing q with state QUERY TARGETS STATE
Mar 31 09:42:36 unbound[3903467:1] info: processQueryTargets: mirrors.fedoraproject.org. A IN
Mar 31 09:42:36 unbound[3903467:1] debug: processQueryTargets: targetqueries 0, currentqueries 0 sentcount 0
Mar 31 09:42:36 unbound[3903467:1] info: DelegationPoint<.>: 13 names (0 missing), 26 addrs (26 result, 0 avail) cacheNS
Mar 31 09:42:36 unbound[3903467:1] info: f.root-servers.net. * A AAAA
Mar 31 09:42:36 unbound[3903467:1] info: auth_zone . query fedoraproject.org. A, domain NULL notexact notexist, ce NULL, rrset NULL
Mar 31 09:42:36 unbound[3903467:1] debug: attempt to get extra 3 targets
Mar 31 09:42:36 unbound[3903467:1] debug: servselect ip4 192.5.5.241 port 53 (len 16)
Mar 31 09:42:36 unbound[3903467:1] debug: rtt=163
Mar 31 09:42:36 unbound[3903467:1] debug: ip6 2001:503:c27::2:30 port 53 (len 28)
Mar 31 09:42:36 unbound[3903467:1] debug: ip4 192.58.128.30 port 53 (len 16)
Mar 31 09:42:36 unbound[3903467:1] debug: ip6 2001:7fd::1 port 53 (len 28)
Mar 31 09:42:36 unbound[3903467:1] debug: ip4 193.0.14.129 port 53 (len 16)
Mar 31 09:42:36 unbound[3903467:1] debug: ip6 2001:500:12::d0d port 53 (len 28)
Mar 31 09:42:36 unbound[3903467:1] debug: ip4 192.112.36.4 port 53 (len 16)
Mar 31 09:42:36 unbound[3903467:1] debug: ip6 2001:dc3::35 port 53 (len 28)
Mar 31 09:42:36 unbound[3903467:1] debug: ip4 202.12.27.33 port 53 (len 16)
Mar 31 09:42:36 unbound[3903467:1] debug: ip6 2001:500:2f::f port 53 (len 28)
Mar 31 09:42:36 unbound[3903467:1] debug: ip4 192.5.5.241 port 53 (len 16)
Mar 31 09:42:36 unbound[3903467:1] debug: iter_handle processing q with state INIT REQUEST STATE (stage 2)
Mar 31 09:42:36 unbound[3903467:1] info: resolving (init part 2): mirrors.fedoraproject.org. AAAA IN
Mar 31 09:42:36 unbound[3903467:1] debug: iter_handle processing q with state INIT REQUEST STATE (stage 3)
Mar 31 09:42:36 unbound[3903467:1] info: resolving (init part 3): mirrors.fedoraproject.org. AAAA IN
Mar 31 09:42:36 unbound[3903467:1] info: schedule dnskey prefetch . DNSKEY IN
Mar 31 09:42:36 unbound[3903467:1] debug: iter_handle processing q with state QUERY TARGETS STATE
Mar 31 09:42:36 unbound[3903467:1] info: processQueryTargets: mirrors.fedoraproject.org. AAAA IN
Mar 31 09:42:36 unbound[3903467:1] debug: processQueryTargets: targetqueries 0, currentqueries 0 sentcount 0
Mar 31 09:42:36 unbound[3903467:1] info: DelegationPoint<.>: empty
Mar 31 09:42:36 unbound[3903467:1] debug: removing 2 labels
Mar 31 09:42:36 unbound[3903467:1] info: auth_zone . query org. A, domain NULL notexact notexist, ce NULL, rrset NULL
Mar 31 09:42:36 unbound[3903467:1] debug: iter_handle processing q with state INIT REQUEST STATE
Mar 31 09:42:36 unbound[3903467:1] info: resolving mirrors.fedoraproject.org. AAAA IN
Mar 31 09:42:36 unbound[3903467:1] debug: request has dependency depth of 0
Mar 31 09:42:36 unbound[3903467:1] debug: cache delegation returns delegpt
Mar 31 09:42:36 unbound[3903467:1] info: DelegationPoint<.>: 13 names (0 missing), 26 addrs (0 result, 26 avail) cacheNS
Mar 31 09:42:36 unbound[3903467:1] info: sending query: fedoraproject.org. A IN
Mar 31 09:42:36 unbound[3903467:1] debug: sending to target: <.> 2001:500:1::53#53
Mar 31 09:42:36 unbound[3903467:1] debug: dnssec status: expected
Mar 31 09:42:36 unbound[3903467:1] debug: EDNS lookup known=0 vs=0
Mar 31 09:42:36 unbound[3903467:1] debug: serviced query UDP timeout=376 msec
Mar 31 09:42:36 unbound[3903467:1] debug: inserted new pending reply id=32fb
Mar 31 09:42:36 unbound[3903467:1] debug: opened UDP if=0 port=56944
Mar 31 09:42:36 unbound[3903467:1] error: udp connect failed: Network is unreachable for 2001:500:1::53 port 53 (len 28)
Mar 31 09:42:36 unbound[3903467:1] info: error sending query to auth server ip6 2001:500:1::53 port 53 (len 28)
Mar 31 09:42:36 unbound[3903467:1] debug: iter_handle processing q with state QUERY TARGETS STATE
Mar 31 09:42:36 unbound[3903467:1] info: processQueryTargets: mirrors.fedoraproject.org. AAAA IN
Mar 31 09:42:36 unbound[3903467:1] debug: processQueryTargets: targetqueries 0, currentqueries 0 sentcount 0
Mar 31 09:42:36 unbound[3903467:1] info: DelegationPoint<.>: 13 names (0 missing), 26 addrs (26 result, 0 avail) cacheNS
Mar 31 09:42:36 unbound[3903467:1] info: f.root-servers.net. * A AAAA
Mar 31 09:42:36 unbound[3903467:1] info: m.root-servers.net. * A AAAA
Mar 31 09:42:36 unbound[3903467:1] info: g.root-servers.net. * A AAAA
Mar 31 09:42:36 unbound[3903467:1] info: k.root-servers.net. * A AAAA
Mar 31 09:42:36 unbound[3903467:1] info: j.root-servers.net. * A AAAA
Mar 31 09:42:36 unbound[3903467:1] info: b.root-servers.net. * A AAAA
Mar 31 09:42:36 unbound[3903467:1] info: c.root-servers.net. * A AAAA
Mar 31 09:42:36 unbound[3903467:1] info: d.root-servers.net. * A AAAA
Mar 31 09:42:36 unbound[3903467:1] info: a.root-servers.net. * A AAAA
Mar 31 09:42:36 unbound[3903467:1] info: i.root-servers.net. * A AAAA
Mar 31 09:42:36 unbound[3903467:1] info: l.root-servers.net. * A AAAA
Mar 31 09:42:36 unbound[3903467:1] info: h.root-servers.net. * A AAAA
Mar 31 09:42:36 unbound[3903467:1] info: e.root-servers.net. * A AAAA
and then:
Mar 31 10:12:01 unbound[3909274:3] info: validation failure <mirrors.fedoraproject.org. AAAA IN>: No DNSKEY record for key fedoraproject.org. while building chain of trust
logs from tcpdump -n -s 1500 udp and port 53 -v:
10:14:59.871494 IP (tos 0x0, ttl 64, id 36015, offset 0, flags [none], proto UDP (17), length 82)
x.x.x.x.57451 > 85.236.55.10.domain: 21341% [1au] A? mirrors.fedoraproject.org. (54)
10:14:59.871670 IP (tos 0x0, ttl 64, id 38142, offset 0, flags [none], proto UDP (17), length 74)
x.x.x.x.39429 > 38.145.60.13.domain: 51344% [1au] DNSKEY? fedoraproject.org. (46)
10:14:59.872004 IP (tos 0x0, ttl 64, id 38143, offset 0, flags [none], proto UDP (17), length 82)
x.x.x.x.37014 > 38.145.60.13.domain: 29471% [1au] AAAA? mirrors.fedoraproject.org. (54)
10:14:59.904828 IP (tos 0x0, ttl 56, id 54889, offset 0, flags [none], proto UDP (17), length 74)
38.145.60.13.domain > x.x.x.x.39429: 51344*-| 0/0/1 (46)
10:14:59.905754 IP (tos 0x0, ttl 56, id 54890, offset 0, flags [none], proto UDP (17), length 917)
38.145.60.13.domain > x.x.x.x.37014: 29471*- 11/0/1 mirrors.fedoraproject.org. CNAME wildcard.fedoraproject.org., mirrors.fedoraproject.org. RRSIG, mirrors.fedoraproject.org. RRSIG, wildcard.fedoraproject.org. AAAA 2605:bc80:3010:600:dead:beef:cafe:fed9, wildcard.fedoraproject.org. AAAA 2604:1580:fe00:0:dead:beef:cafe:fed1, wildcard.fedoraproject.org. AAAA 2620:52:3:1:dead:beef:cafe:fed7, wildcard.fedoraproject.org. AAAA 2605:bc80:3010:600:dead:beef:cafe:feda, wildcard.fedoraproject.org. AAAA 2610:28:3090:3001:dead:beef:cafe:fed3, wildcard.fedoraproject.org. AAAA 2620:52:3:1:dead:beef:cafe:fed6, wildcard.fedoraproject.org. RRSIG, wildcard.fedoraproject.org. RRSIG (889)
10:14:59.906008 IP (tos 0x0, ttl 64, id 31976, offset 0, flags [none], proto UDP (17), length 83)
x.x.x.x.49864 > 38.145.60.14.domain: 62491% [1au] A? wildcard.fedoraproject.org. (55)
10:14:59.906196 IP (tos 0x0, ttl 64, id 36016, offset 0, flags [none], proto UDP (17), length 74)
x.x.x.x.53032 > 85.236.55.10.domain: 45398% [1au] DNSKEY? fedoraproject.org. (46)
10:14:59.917198 IP (tos 0x0, ttl 56, id 453, offset 0, flags [none], proto UDP (17), length 565)
38.145.60.14.domain > x.x.x.x.49864: 62491*- 12/0/1 wildcard.fedoraproject.org. A 152.19.134.198, wildcard.fedoraproject.org. A 38.145.60.20, wildcard.fedoraproject.org. A 8.43.85.67, wildcard.fedoraproject.org. A 8.43.85.73, wildcard.fedoraproject.org. A 67.219.144.68, wildcard.fedoraproject.org. A 152.19.134.142, wildcard.fedoraproject.org. A 209.132.190.2, wildcard.fedoraproject.org. A 140.211.169.196, wildcard.fedoraproject.org. A 140.211.169.206, wildcard.fedoraproject.org. A 38.145.60.21, wildcard.fedoraproject.org. RRSIG, wildcard.fedoraproject.org. RRSIG (537)
10:14:59.917439 IP (tos 0x0, ttl 64, id 38157, offset 0, flags [none], proto UDP (17), length 83)
x.x.x.x.41571 > 38.145.60.13.domain: 38290% [1au] AAAA? wildcard.fedoraproject.org. (55)
10:14:59.926376 IP (tos 0x0, ttl 56, id 54909, offset 0, flags [none], proto UDP (17), length 573)
38.145.60.13.domain > x.x.x.x.41571: 38290*- 8/0/1 wildcard.fedoraproject.org. AAAA 2620:52:3:1:dead:beef:cafe:fed7, wildcard.fedoraproject.org. AAAA 2605:bc80:3010:600:dead:beef:cafe:fed9, wildcard.fedoraproject.org. AAAA 2604:1580:fe00:0:dead:beef:cafe:fed1, wildcard.fedoraproject.org. AAAA 2605:bc80:3010:600:dead:beef:cafe:feda, wildcard.fedoraproject.org. AAAA 2620:52:3:1:dead:beef:cafe:fed6, wildcard.fedoraproject.org. AAAA 2610:28:3090:3001:dead:beef:cafe:fed3, wildcard.fedoraproject.org. RRSIG, wildcard.fedoraproject.org. RRSIG (545)
10:14:59.959706 IP (tos 0x0, ttl 56, id 43304, offset 0, flags [none], proto UDP (17), length 909)
85.236.55.10.domain > x.x.x.x.57451: 21341*- 15/0/1 mirrors.fedoraproject.org. CNAME wildcard.fedoraproject.org., mirrors.fedoraproject.org. RRSIG, mirrors.fedoraproject.org. RRSIG, wildcard.fedoraproject.org. A 67.219.144.68, wildcard.fedoraproject.org. A 38.145.60.20, wildcard.fedoraproject.org. A 140.211.169.196, wildcard.fedoraproject.org. A 38.145.60.21, wildcard.fedoraproject.org. A 209.132.190.2, wildcard.fedoraproject.org. A 8.43.85.73, wildcard.fedoraproject.org. A 8.43.85.67, wildcard.fedoraproject.org. A 152.19.134.198, wildcard.fedoraproject.org. A 152.19.134.142, wildcard.fedoraproject.org. A 140.211.169.206, wildcard.fedoraproject.org. RRSIG, wildcard.fedoraproject.org. RRSIG (881)
10:14:59.960064 IP (tos 0x0, ttl 64, id 32004, offset 0, flags [none], proto UDP (17), length 83)
x.x.x.x.60137 > 38.145.60.14.domain: 46187% [1au] A? wildcard.fedoraproject.org. (55)
10:14:59.968029 IP (tos 0x0, ttl 55, id 465, offset 0, flags [none], proto UDP (17), length 565)
38.145.60.14.domain > x.x.x.x.60137: 46187*- 12/0/1 wildcard.fedoraproject.org. A 38.145.60.21, wildcard.fedoraproject.org. A 8.43.85.67, wildcard.fedoraproject.org. A 38.145.60.20, wildcard.fedoraproject.org. A 140.211.169.196, wildcard.fedoraproject.org. A 140.211.169.206, wildcard.fedoraproject.org. A 152.19.134.198, wildcard.fedoraproject.org. A 152.19.134.142, wildcard.fedoraproject.org. A 209.132.190.2, wildcard.fedoraproject.org. A 8.43.85.73, wildcard.fedoraproject.org. A 67.219.144.68, wildcard.fedoraproject.org. RRSIG, wildcard.fedoraproject.org. RRSIG (537)
10:14:59.994171 IP (tos 0x0, ttl 56, id 43315, offset 0, flags [none], proto UDP (17), length 74)
85.236.55.10.domain > x.x.x.x.53032: 45398*-| 0/0/1 (46)
10:14:59.995502 IP (tos 0x0, ttl 64, id 63925, offset 0, flags [none], proto UDP (17), length 74)
x.x.x.x.60014 > 152.19.134.139.domain: 34916% [1au] DNSKEY? fedoraproject.org. (46)
The log snippet you show only shows the lack of IPv6 network, and that is not a concern here. You need to dig into the logs to see where the lookups fail (for other reasons than IPv6 network not connected).
My guess is that the fedoraproject.org DNSKEY query is failing. It asks three times in the snippet you give. And it does not give answers. Can you try that with dig commands? Perhaps you have PMTU problems, the packet is too big to fit in an UDP reply to you, such large packets are not allowed or possible near your machine on the network. And this causes TCP fallback, to fetch it over TCP if it manages to signal that it is too big, so it could be the issue.
OK I think we are getting somewhere. I posted a debug log for the duration of the failing dnf here. Note when I set val-permissive-mode to yesdnf update works.
building chain of trust
Mar 31 11:33:53 unbound[3925918:1] debug: mesh_run: validator module exit state is module_finished
Mar 31 11:33:53 unbound[3925918:1] debug: ipsecmod[module 0] operate: extstate:module_state_initial event:module_event_moddone
Mar 31 11:33:53 unbound[3925918:1] debug: mesh_run: ipsecmod module exit state is module_finished
Mar 31 11:33:53 unbound[3925918:1] debug: validator[module 1] operate: extstate:module_wait_subquery event:module_event_pass
Mar 31 11:33:53 unbound[3925918:1] info: validator operate: query mirrors.fedoraproject.org. AAAA IN
Mar 31 11:33:53 unbound[3925918:1] debug: val handle processing q with state VAL_VALIDATE_STATE
Mar 31 11:33:53 unbound[3925918:1] info: Could not establish a chain of trust to keys for fedoraproject.org. DNSKEY IN
Mar 31 11:33:53 unbound[3925918:1] debug: val handle processing q with state VAL_FINISHED_STATE
Mar 31 11:33:53 unbound[3925918:1] info: validation failure <mirrors.fedoraproject.org. AAAA IN>: No DNSKEY record for key fedoraproject.org. while building chain of trust
Mar 31 11:33:53 unbound[3925918:1] debug: mesh_run: validator module exit state is module_finished
Mar 31 11:33:53 unbound[3925918:1] debug: ipsecmod[module 0] operate: extstate:module_wait_module event:module_event_moddone
Mar 31 11:33:53 unbound[3925918:1] info: ipsecmod operate: query mirrors.fedoraproject.org. AAAA IN
Mar 31 11:33:53 unbound[3925918:1] debug: mesh_run: ipsecmod module exit state is module_finished
Mar 31 11:33:53 unbound[3925918:1] info: send_udp over interface: 1 127.0.0.1 127.0.0.1
Mar 31 11:33:53 unbound[3925918:1] debug: query took 7.962752 sec
Mar 31 11:33:53 unbound[3925918:1] reply: 127.0.0.1 mirrors.fedoraproject.org. AAAA IN SERVFAIL 7.962752 0 43
Mar 31 11:33:53 unbound[3925918:1] info: mesh_run: end 0 recursion states (0 with reply, 0 detached), 0 waiting replies, 4 recursion replies sent, 0 replies dropped, 0 states jostled out
Mar 31 11:33:53 unbound[3925918:1] info: average recursion processing time 3.760770 sec
Mar 31 11:33:53 unbound[3925918:1] info: histogram of recursion processing times
Mar 31 11:33:53 unbound[3925918:1] info: [25%]=0.196608 median[50%]=0.262144 [75%]=6
Mar 31 11:33:53 unbound[3925918:1] info: lower(secs) upper(secs) recursions
Mar 31 11:33:53 unbound[3925918:1] info: 0.131072 0.262144 2
Mar 31 11:33:53 unbound[3925918:1] info: 4.000000 8.000000 2
Mar 31 11:33:53 unbound[3925918:1] debug: cache memory msg=144072 rrset=157024 infra=47892 val=80250 subnet=0
Looks like that DNSKEY lookup is not working. Try digging for the dig fedoraproject.org DNSKEY +dnssec and see how that works out; that is what seems to be failing for unbound. Looks like PMTU and no TCP fallback possible because TCP is blocked by firewall.
Yes looks like it's failing. I tested do-tcp: no, same problem. Did you see the debug logs? It's a ping-pong match, going back and forth between UDP and TCP.
No, I mean do that dig to the ip address of one of the nameservers for fedoraproject.org. Not to the unbound like you just did.
If so, that it fails to work, then it seems the PMTU size is a problem. What edns-buffer-size do you have configured? Consider lowering it to 1232, and that should make fedoraproject.org return truncated. But then unbound has to use TCP to fetch it and you have no TCP access. If you fix the TCP connectivity it then works, I guess.
I'm not sure what you are asking me to do. Below are dig's from another server not using unbound to both the NS and IP for fedoraproject.org. As for edns-buffer-size it still is commented out to the default 1232:
# EDNS reassembly buffer to advertise to UDP peers (the actual buffer
# is set with msg-buffer-size). 1472 can solve fragmentation (timeouts)
# edns-buffer-size: 1232
dig @152.19.134.139 fedoraproject.org DNSKEY +dnssec is what I wanted, that is similar to what unbound does itself.
From your logs it looks like that fails. It tries to signal via UDP that it is truncated. Your edns-buffer-size looks okay. And the reason that this lookup fails is then because TCP fails for you. The dig command should show that.
You then have the proof what query fails and why. I guess you need to fix TCP connectivity.
See below. What does fix this though is if I set val-permissive-mode to yes everything including dnf update works. My colleagues believe the issue is related to DNSSEC.
So in your logs, that was tcpdump, unbound keeps asking for this but cannot get an answer. But when you dig for it, it works fine. Can you get a verbosity 4 log of it so I can see what unbound is doing, and getting as a reply? Because that looks different and not like what dig got.
Oh I just noticed that dig gets edns-buffer-size: 4096 . Not sure what it set in the query, it may also have set that. In that case edns-buffer-size: 4096 may fix your lookups. Because unbound has 1232, which is a better value because of fragmentation, but causes TCP fallback. And TCP does not work for this server. So 4096 would make it likely work without TCP fallback.
It would still be a better recommendation to have edns-buffer-size: 1232 (the default, so no config line needed), and fix so that TCP connectivity works for this server. That is better for UDP fragmentation and then lookups work too, using the TCP lookups when needed. But likely the edns-buffer-size setting above could fix your immediate problem.
Oh I just noticed that dig gets edns-buffer-size: 4096 . Not sure what it set in the query, it may also have set that. In that case edns-buffer-size: 4096 may fix your lookups. Because unbound has 1232, which is a better value because of fragmentation, but causes TCP fallback. And TCP does not work for this server. So 4096 would make it likely work without TCP fallback.
Indeed setting edns-buffer-size: 4096 fixes the dnf updates with fedoraproject.org! So in this case TCP fallback isn't needed. I will continue to follow up with our IT group on allowing 53/TCP.
What issues might be experienced with having edns-buffer-size: 4096?
Fragmentation issues, these can be network issues but also reassembly can get exploited. It was the topic of DNS Flag Day, https://dnsflagday.net/2020/ .
Seems that this issue may be closed.
Please don't hijack issues: This was about a file permission error upon service start, not firewalling issues upon service usage.
Activity
wcawijngaards commentedon Mar 19, 2021
Unbound periodically writes to the root.key file, to process the updates for RFC5011 updates. That is what you have configured.
The error is that unbound wants to create a temporary file to write the new contents of the /var/lib/unbound/root.key into. After that unbound is going to remove the old root.key file and rename this file to the root.key file. But unbound gets permission denied when it attempts to write there. Perhaps the directory has no write permissions (for the unbound user), or you have selinux or apparmor or freebsd capabilities configured to deny write access from unbound to this directory?
LinuxPersonEC commentedon Mar 19, 2021
I try to
chown unbound:unboundbut after restarting the unbound process it goes back to root;No SELinux, no AppArmor, and you can see from my rpm command it's Fedora 33.
Now I see:
fatal error: could not open autotrust file for writing, /var/lib/unbound/root.key.2653680-3-56517c61b7d0: Permission deniedLinuxPersonEC commentedon Mar 19, 2021
Not sure if this is the long term fix but I ran
chown -R unbound:unbound /var/lib/unboundand restarted unbound but I get theseerror: read (in tcp s): Connection refusedas mentioned in issue 360.TCP Fast Open is disabled. How can I debug this?
Edit I see a user having this with Netgate.
- Fix for #447: squelch connection refused tcp connection failures
wcawijngaards commentedon Mar 19, 2021
The chown of the directory worked, and that fixed your permission denied failure that you had.
The connection refused, is a spammy message that says unbound cannot make tcp connections. I have made it less visible, by only making it print when verbosity is high. This is committed to the repository.
It seems you cannot make TCP connections, and dig also cannot make tcp connections to those IP addresses from the commandline. I also do not know why the connections are refused by your upstream. Perhaps your firewall is refusing to allow tcp connections there, or another firewall is refusing tcp connections. This is a separate issue from your permission denied issue.
LinuxPersonEC commentedon Mar 19, 2021
Once I removed the
-doption from/usr/lib/systemd/system/unbound.serviceinExecStartthe TCP timeout issues also disappeared from the logs. Perhaps this was already done upstream?Anyways, as for the original issue, yes it's fixed. I haven't tried a reboot but I'm assuming it should be persistent. But I don't see this documented anywhere to change the permission in the
/var/lib/unbounddirectory. Should that be part of the installation process?wcawijngaards commentedon Mar 22, 2021
Not sure how the removal of the -d option fixes your issue, perhaps what happened was you reloaded your new config file in the process with fixed options in it? Or it fixed it somehow, the unbound.service file is part of the unbound rpm that you are using.
Also the var/lib/unbound directory is created by the rpm and also the configuration that unbound uses that directory to store files is part of the rpm configure script (or default configuration shipped with the package). But we do not do the package scripts, that is handled by the distro that you are using. A look at the unbound.spec file shows that it performs a chown for the user unbound on install
%dir %attr(0755,unbound,unbound) %{_sharedstatedir}/%{name}, so I do not know what is wrong with that.LinuxPersonEC commentedon Mar 25, 2021
I can confirm that on a second server running unbound, that when I removed the
-doption from the/usr/lib/systemd/system/unbound.serviceall of thoseerror: read (in tcp s): Connection refusederrors disappeared.wcawijngaards commentedon Mar 25, 2021
The -d option, documented with
-d do not fork into the background., makes unbound not fork into a new process. So with-dit stays attached to the console on which it was started. This is what systemd wants because systemd is monitoring that console.Systemd may also have set other permissions on that console, did it? Would that have caused the different results for tcp connections for the program? Without the
-doption unbound forks into a process, but I do not really see what difference that could have made to the tcp connections, so that it gives no connection refused.LinuxPersonEC commentedon Mar 29, 2021
Can you clarify what you are looking for? Here is what is in the
/usr/lib/systemd/system/unbound.servicefile.wcawijngaards commentedon Mar 29, 2021
From this I see no reason why
-dwould have made a difference to TCP connections.LinuxPersonEC commentedon Mar 29, 2021
After I removed our DNS servers in /etc/resolv.conf unbound stops working:
Port 8953 is open. What else is wrong?
LinuxPersonEC commentedon Mar 29, 2021
OK a little embarrassing here. The
-doption is definitely needed.interface-automaticwas set to the default,no.I'm still getting the
error: read (in tcp s): Connection refused forerrors but at leastunboundis working.wcawijngaards commentedon Mar 30, 2021
So dig fails to make a tcp connection. Unbound also fails to make that tcp connection. Your system does not make tcp connections.
Perhaps a firewall issue, on this device or on another device?
LinuxPersonEC commentedon Mar 31, 2021
Indeed our IT group confirmed they started blocking all non-UDP requests to DNS. They are looking at allowing it back just for our department.
You previously mentioned TCP Fast Open. Does unbound take advantage of it and any place I can share documentation for it?
Also if TCP is blocked for IPs outside of our subnet, would that cause dnf update to fail with the following?
3 remaining items
LinuxPersonEC commentedon Mar 31, 2021
Perhaps it's not TCP but DNSSEC that is expected? See logs.
and then:
Mar 31 10:12:01 unbound[3909274:3] info: validation failure <mirrors.fedoraproject.org. AAAA IN>: No DNSKEY record for key fedoraproject.org. while building chain of trustlogs from
tcpdump -n -s 1500 udp and port 53 -v:Edit: DNSSEC test:
wcawijngaards commentedon Mar 31, 2021
The log snippet you show only shows the lack of IPv6 network, and that is not a concern here. You need to dig into the logs to see where the lookups fail (for other reasons than IPv6 network not connected).
My guess is that the fedoraproject.org DNSKEY query is failing. It asks three times in the snippet you give. And it does not give answers. Can you try that with dig commands? Perhaps you have PMTU problems, the packet is too big to fit in an UDP reply to you, such large packets are not allowed or possible near your machine on the network. And this causes TCP fallback, to fetch it over TCP if it manages to signal that it is too big, so it could be the issue.
LinuxPersonEC commentedon Mar 31, 2021
OK I think we are getting somewhere. I posted a debug log for the duration of the failing dnf here. Note when I set
val-permissive-modetoyesdnf updateworks.wcawijngaards commentedon Apr 1, 2021
Looks like that DNSKEY lookup is not working. Try digging for the dig fedoraproject.org DNSKEY +dnssec and see how that works out; that is what seems to be failing for unbound. Looks like PMTU and no TCP fallback possible because TCP is blocked by firewall.
LinuxPersonEC commentedon Apr 1, 2021
Yes looks like it's failing. I tested
do-tcp: no, same problem. Did you see the debug logs? It's a ping-pong match, going back and forth between UDP and TCP.wcawijngaards commentedon Apr 1, 2021
No, I mean do that dig to the ip address of one of the nameservers for fedoraproject.org. Not to the unbound like you just did.
If so, that it fails to work, then it seems the PMTU size is a problem. What edns-buffer-size do you have configured? Consider lowering it to 1232, and that should make fedoraproject.org return truncated. But then unbound has to use TCP to fetch it and you have no TCP access. If you fix the TCP connectivity it then works, I guess.
LinuxPersonEC commentedon Apr 1, 2021
I'm not sure what you are asking me to do. Below are dig's from another server not using unbound to both the NS and IP for fedoraproject.org. As for
edns-buffer-sizeit still is commented out to the default 1232:wcawijngaards commentedon Apr 1, 2021
dig @152.19.134.139 fedoraproject.org DNSKEY +dnssecis what I wanted, that is similar to what unbound does itself.From your logs it looks like that fails. It tries to signal via UDP that it is truncated. Your edns-buffer-size looks okay. And the reason that this lookup fails is then because TCP fails for you. The dig command should show that.
You then have the proof what query fails and why. I guess you need to fix TCP connectivity.
LinuxPersonEC commentedon Apr 1, 2021
See below. What does fix this though is if I set
val-permissive-modetoyeseverything includingdnf updateworks. My colleagues believe the issue is related to DNSSEC.wcawijngaards commentedon Apr 1, 2021
So in your logs, that was tcpdump, unbound keeps asking for this but cannot get an answer. But when you dig for it, it works fine. Can you get a verbosity 4 log of it so I can see what unbound is doing, and getting as a reply? Because that looks different and not like what dig got.
Oh I just noticed that dig gets edns-buffer-size: 4096 . Not sure what it set in the query, it may also have set that. In that case edns-buffer-size: 4096 may fix your lookups. Because unbound has 1232, which is a better value because of fragmentation, but causes TCP fallback. And TCP does not work for this server. So 4096 would make it likely work without TCP fallback.
It would still be a better recommendation to have edns-buffer-size: 1232 (the default, so no config line needed), and fix so that TCP connectivity works for this server. That is better for UDP fragmentation and then lookups work too, using the TCP lookups when needed. But likely the edns-buffer-size setting above could fix your immediate problem.
LinuxPersonEC commentedon Apr 1, 2021
Indeed setting
edns-buffer-size: 4096fixes thednf updateswith fedoraproject.org! So in this case TCP fallback isn't needed. I will continue to follow up with our IT group on allowing 53/TCP.What issues might be experienced with having
edns-buffer-size: 4096?wcawijngaards commentedon Apr 1, 2021
Fragmentation issues, these can be network issues but also reassembly can get exploited. It was the topic of DNS Flag Day, https://dnsflagday.net/2020/ .
nursoda commentedon Jan 5, 2022
Seems that this issue may be closed.
Please don't hijack issues: This was about a file permission error upon service start, not firewalling issues upon service usage.
wcawijngaards commentedon Jan 5, 2022
Sure, it seems that the issues have been resolved.