DNS-01: change default settings to avoid negative-cache delay and timeouts?

[csmith]

In lego v5, checking TXT records using the recursive resolvers was enabled by default.

Combined with CNAME checking (also enabled by default), this seems to create an issue if the resolvers cache NXDOMAIN responses across all qtypes (which is what they "should" do per RFC2308).

From what I can see, the relevant DNS queries when performing a DNS-01 challenge are:

1. Query CNAME to determine where to create the record
2. Create the TXT record through the provider
3. Query TXT recursively to discover CNAMEs (tolerates NXDOMAIN)
4. Check the expected TXT value on every configured recursive resolver (waiting until it's present)
5. Discover the authoritative servers and check TXT directly there (waiting until it's present)

If the challenge name doesn't already exist, the CNAME query at 1 will result in an NXDOMAIN. Resolvers like systemd-resolved will keep this cached for min(SOA TTL, SOA minimum). The TXT query at 4 then receives this cached response, delaying the validation until the cache expires, or failing issuance entirely if the remaining negative-cache lifetime exceeds the provider’s propagation timeout.

Disabling the recursive propagation requirement with LEGO_DNS_PROPAGATION_DISABLE_RNS=true avoids this particular wait while retaining authoritative checks. Reducing the zone’s negative-cache TTL can also mitigate it. Disabling CNAME support may help, but providers can also perform SOA lookups before creating the record, potentially seeding the same negative cache. But with the defaults as they are, fairly normal DNS configurations can encounter unnecessary delays or timeouts.

It looks like the recursive resolver check was added to deal with a use case where lego checks the same recursive resolvers as the CA, so I'm wondering if just flipping the default back to off would make sense for most users?

[ECD5A]

I'd test the default change with two separate DNS states: cached NXDOMAIN at the recursive resolver while the authoritative server already has the correct TXT, and an uncached name where the authoritative server still lacks the TXT. With the recursive requirement disabled and authoritative checks enabled, the first should proceed and the second should keep waiting.

The shared-resolver case from #2276 is the tradeoff: passing the authoritative check doesn't mean the CA's recursive cache has expired. Disabling the requirement can move that failure from lego's precheck to ACME validation, so I'd keep explicit recursive checking available for that setup.

Also, the current precheck still performs the recursive CNAME lookup when the recursive TXT requirement is disabled. A regression test should retain that lookup so it covers the actual query sequence you described.

[ldez]

I will not change the default, at least not in the v5 because this is a breaking change, and I don't plan to create a v6 for now.