cache invalidation issue with CNAME+A

[slayer1366]

Let's assume that we have the following record in zone zone.net, authoritatitve name server responses correctly in such way

dig @ns1.zone.net c-c9qah5rl9qm58427p78r.rw.db.zone.net. -t A
c-c9qah5rl9qm58427p78r.rw.db.zone.net. 10 IN CNAME rc1a-ty9dwtfubb3dq8ti.db.zone.net.
rc1a-ty9dwtfubb3dq8ti.db.zone.net. 300 IN A 178.154.226.240

Also we have recursive unbound that could return (from cache) for A question either eventually only CNAME or CNAME and A resolved (as expected). Steps to reproduce: against recursive unbound ::1 do in cycle:

dig @::1 c-c9qah5rl9qm58427p78r.rw.db.zone.net. -t CNAME && dig @::1 c-c9qah5rl9qm58427p78r.rw.db.zone.net. -t A

response for -t CNAME (it always OK)
; QUESTION SECTION:
c-c9qah5rl9qm58427p78r.rw.db.zone.net. IN CNAME
;; ANSWER SECTION:
c-c9qah5rl9qm58427p78r.rw.db.zone.net. 5 IN CNAME rc1a-ty9dwtfubb3dq8ti.db.zone.net.

response for -t A (it sometimes ok, but not always) here could be two possible answers:

1. incorrect answer (please note that we request A type, expecting that unbound will resolve CNAME and than A in chain to return A record also), it could return this kind of replies eventually, I suppose it correlates with cache ttl time
;; QUESTION SECTION:
;c-c9qah5rl9qm58427p78r.rw.db.zone.net. IN A
;; ANSWER SECTION:
c-c9qah5rl9qm58427p78r.rw.db.zone.net. 5 IN CNAME rc1a-ty9dwtfubb3dq8ti.db.zone.net.
;; AUTHORITY SECTION:
zone.net. 81820 IN NS ns1.zone.net.
zone.net. 81820 IN NS ns1.zone.net.

2. correct answer
;; QUESTION SECTION:
;c-c9qah5rl9qm58427p78r.rw.db.zone.net. IN A
;; ANSWER SECTION:
c-c9qah5rl9qm58427p78r.rw.db.zone.net. 12 IN CNAME rc1a-ty9dwtfubb3dq8ti.db.zone.net.
rc1a-ty9dwtfubb3dq8ti.db.zone.net. 178 IN A 178.154.226.240

It seems that CNAME request for record c-c9qah5rl9qm58427p78r.rw.db.zone.net. somehow invalidates cache with incorrect answer and unbound tries to use it for question with A also.

Unbound versions checked and affected 1.10.1, 1.13.0.

[wcawijngaards]

What is likely happening is that Unbound tries to lookup the A record again, but cannot fetch it. The upstream server responds without the A record in the answer, but instead includes the zone.net NS records. This is a noerror/nodata answer or perhaps an nxdomain answer that is a little mangled. If the upstream server decides the answer to the A query is nothing, then unbound copies that.

You can check what is going on by setting verbosity higher, like 4 or 5 or so, that makes the server slower in that it prints everything, but it also prints the response packets from upstream servers. An in this case, the response packet when unbound asks the upstream server for the rc1a-.. A record. That response is likely different from time to time. And this causes unbound to give a different answer.

Unbound does not create those NS ns1.zone.net contents by itself, so it must have had some sort of input from the upstream server for that. That must have excluded the A record.

If that did include the A record, there is a setting, private-address, if the upstream server creates varying A record with different contents, the private-address is a feature where unbound removes A record with particular netblocks. If you have that configured, then unbound could be performing that feature and removing the A record when it is different and matches the netblocks that have to be removed. Some people use this feature because they do not want other networks to talk about their IP space.

[slayer1366]

dns dump shows only two types of requests

14:42:24.781553 IP6 ... 82) 2a02:...433f:1:1.9981 > 2a02..1001.53: [bad udp cksum 0xc22a -> 0x4df3!] 35750% [1au] A? c-c9qah5rl9qm58427p78r.rw.db.zone.net. (74)
14:42:24.781759 IP6 ... 213) 2a02...1001.53 > 2a02...433f:1:1.9981: [udp sum ok] 35750*- 2/2/1 c-c9qah5rl9qm58427p78r.rw.db.zone.net. CNAME rc1a-ty9dwtfubb3dq8ti.db.zone.net., rc1a-ty9dwtfubb3dq8ti.db.zone.net. A 178.154.226.240 (205)

14:42:24.781815 IP6 ... 82) 2a02...433f:1:1.35071 > 2a02...1001.53: [bad udp cksum 0xc22a -> 0xe235!] 37217% [1au] CNAME? c-c9qah5rl9qm58427p78r.rw.db.zone.net. (74)
14:42:24.782183 IP6 ... 197) 2a02:...1001.53 > 2a02:.433f:1:1.35071: [udp sum ok] 37217*- 1/2/1 c-c9qah5rl9qm58427p78r.rw.db.zone.net. CNAME rc1a-ty9dwtfubb3dq8ti.db.zone.net. (189)

so authoritative servers response as expected. I try to increase verbosity upto 4/5.

[wcawijngaards]

So it is not the request for c-c9qah5rl9qm58427p78r.rw.db.zone.net. that I am looking for, but the request for rc1a-ty9dwtfubb3dq8ti.db.zone.net. IN A. Unbound asks for that separately.

[slayer1366]

It could have all information in first request - see

14:42:24.781553 IP6 ... 82) 2a02:...433f:1:1.9981 > 2a02..1001.53: [bad udp cksum 0xc22a -> 0x4df3!] 35750% [1au] A? c-c9qah5rl9qm58427p78r.rw.db.zone.net. (74)

14:42:24.781759 IP6 ... 213) 2a02...1001.53 > 2a02...433f:1:1.9981: [udp sum ok] 35750*- 2/2/1 c-c9qah5rl9qm58427p78r.rw.db.zone.net. CNAME rc1a-ty9dwtfubb3dq8ti.db.zone.net., rc1a-ty9dwtfubb3dq8ti.db.zone.net. A 178.154.226.240 (205)

here - we have knowledge that
rc1a-ty9dwtfubb3dq8ti.db.zone.net. A 178.154.226.240 (205)

no any additional request about A, also A record could be in cache.

[wcawijngaards]

No unbound does not use that A record that is also in the first request. This is necessary to avoid a whole long list of vulnerabilities and issues with indirections. Thus, unbound asks for the components of the CNAME chain by themselves, one by one. The upstream server includes it in the response, but we cannot use it, because of cache poisoning issues. Instead, unbound must ask again.

So unbound should have queried for that rc1a-ty9dwtfubb3dq8ti.db.zone.net A on a query by itself.

[wcawijngaards]

But you are right that the additional request about A could not be there because the A record is still in cache. If that is so, wait until the TTL time expires, and then unbound makes the request and could get the faulty response.

And I note in your first post that the TTL on the bad response is 81820 and that is very high compared to the 178 for the good response. Perhaps it was 86400 (24h) TTL on a negative response, that has counted down for some hours.

[slayer1366]

steps to reproduce: (1) if you constantly asks for -t A - it's okey, (2) if you constantly asks for -t CNAME - it's okey also, but if you could try to simultaneously ask for -t A and -t CNAME you could finish with such kind of problem. I turned verbosity to 5 - but logs seem to be strange. It seems that received answers from CNAME part override the answers for A part. So my guess is that unbound uses answers it received before - in previous requests processing instead of making new requests/lookups and that staff

could I sent to you logs directly to investigate them?

[wcawijngaards]

Unbound has a cache. And uses that. Also if different lookups look up the same information it can share the results between them. But that should not break the CNAME lookup like you shown above; so that should not happen. If you want to send logs certainly that could be useful! Reproducing the problem is very helpful for getting the problem fixed.

[slayer1366]

Thanks a lot, I sent detailed logs about this issue to your email.

[slayer1366]

Wouter, did you receive debug logs? Could you find anything related to this issue?

[dukeartem]

Hello Wouter and slayer1366. I have the same problem and i can reproduce in 100% of case. For this:

1. created A and CNAME records:

dukeartem-cname.zone.net.        10  	IN 	CNAME dukeartem-a.zone.net.
dukeartem-a.zone.net.            300 	IN 	A 127.0.0.1

where CNAME has very small TTL and A has standard TTL
2. changed some options in unbound.conf for more stable reproduce

response-ttl-algo: "default"
        cache-min-ttl: 5
        cache-max-ttl: 10

3. And executed command:

dig @::1 dukeartem-cname.zone.net -t CNAME; dig @::1 dukeartem-cname.zone.net -t A ;

first command return:

; <<>> DiG 9.11.4-P1 <<>> @::1 dukeartem-cname.zone.net -t CNAME
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 8312
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;dukeartem-cname.zone.net.	IN	CNAME

;; ANSWER SECTION:
dukeartem-cname.zone.net.	10 IN	CNAME	dukeartem-a.zone.net.

;; Query time: 11 msec
;; SERVER: ::1#53(::1)
;; WHEN: Mon jan 31 16:56:12 MSK 2022
;; MSG SIZE  rcvd: 85

And that's ok, we get what we want. Second command return:

; <<>> DiG 9.11.4-P1 <<>> @::1 dukeartem-cname.zone.net -t A
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 45877
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;dukeartem-cname.zone.net.	IN	A

;; ANSWER SECTION:
dukeartem-cname.zone.net.	10 IN	CNAME	dukeartem-a.zone.net.

;; AUTHORITY SECTION:
zone.net.		10	IN	NS	ns2.zone.net.
zone.net.		10	IN	NS	ns1.zone.net.

;; Query time: 0 msec
;; SERVER: ::1#53(::1)
;; WHEN: Mon jan 31 16:56:12 MSK 2022
;; MSG SIZE  rcvd: 128

And i get for A request only CNAME record. If i switch dig commands the problem doesn't not reproducing.
Any idea what to do for fix this problem?

[wcawijngaards]

This does not reproduce it for me. It works fine when I try that in a test.

What is the answer to a query, to the authoritative, for dukeartem-a.zone.net -t A ? Or do you have verbose, verbosity 4, logs for what is happening? The cache-max-ttl does not really seem to be doing anything, well it changes the TTL of the answer, but I see no correlation with the issue that is the flaw. Also for cache-min-ttl.

In my test case, unbound picks up the previously created CNAME from cache, and then makes that query to the authoritative. But I see in the answer, that there is an AUTHORITY section with zone.net NS records, so I wonder if there is something amiss with that.

[wcawijngaards]

The issue was that the module-config had to be set, and it was not cache-ttl settings. That reproduced it and it was the qname-minimisation store of an intermediate lookup that was the problem. The patch fixes it and solves the problem. Thanks for the detailed reports!