Test failure rates for CAA queries

[rolandshoemaker]

Currently if a LookupCAA query hits a SERVFAIL response code it will return an empty set of CAA records and continue as normal (which is equivalent to a domain not having any CAA records). This was initially done because a number of large DNS providers did not support CAA and would return a REFUSED response code, instead of say NODATA. Unfortunately Unbound returns the same RCODE, SERVFAIL, if a upstream resolver returns REFUSED as it returns if the answer fails DNSSEC validation.

In the interim Cloudflare seems to have silently fixed this, so we should now run a test against a large sample of domains we have issued for to see if a large enough number of other DNS providers support CAA. If most providers do support CAA we should remove the code that treats a returned SERVFAIL the same as a domain having no records.

[jsha]

I wrote a quick Boulder script to check this: lookups.txt

It's still running, but here are results so far:

 124925 NOERROR
   2644 NXDOMAIN
    335 REFUSED
    348 SERVFAIL

[rolandshoemaker]

Could you run the same sample against a authoritative Unbound instance and see if anything changes? (beyond the REFUSED -> SERVFAIL conversion)

[jsha]

This was against an authoritative Unbound.

[jsha]

Although perhaps not.. I'm also running a copy on a VPS, talking to a local Unbound with the exact same config as prod, and no longer get any REFUSED in the logs. So I may have screwed up the config when generating the preview above.

[rolandshoemaker]

Hm, are you running different versions of Unbound on the two machines perchance?

[jsha]

First: 1.4.22, second: 1.4.20

[riking]

Here's what I got on my scan of the ICANNTLD list, running through BIND:

NOERROR:6428
NXDOMAIN:800
SERVFAIL:28
udp timeout:10

---

Timeout Domains

i.ph
mil
xn--h1aegh.museum
test.tj
travel.tt
res.aero
tsaritsyn.ru
sch.lk
nakhodka.ru
per.la

---

SERVFAIL domains

edu.la
embaixada.st
net.st
mil.tm
cc.id.us
gov.as
store.bb
info.la
com.la
palana.ru
norilsk.ru
consulado.st
mil.st
co.tm
tt.im
asso.km
computerhistory.museum
cc.wa.us
gov.la
org.la
biz.mv
mil.no
fareast.ru
edu.st
store.st
net.la
mil.pe
edu.tm

[jsha]

I tried again with a bigger instance, and went through the list again:

     15 dns: failed to unpack truncated message
   3840 SERVFAIL
  11510 i/o timeout
  83311 NXDOMAIN
2382507 NOERROR

So, 0.1% of currently-issued names respond SERVFAIL for CAA.

Note: When I spot-checked some of the SERVFAIL hostnames, some of them also returned SERVFAIL for A lookups. The next step would be to run a test that looks up both A and CAA together, and counts the result only if the A lookup was successful.

[jsha]

Updated code at https://github.com/jsha/go/blob/master/caa-lookups/lookups.go checks A before CAA.

[jsha]

Next steps on this: get the contact list, notify, document, flip flag.

[jsha]

Ticket filed to enable this in staging.