v2.38.0 — crawler verification with a time limit

v2.38.0

Crawler verification with a time limit.

Reverse-DNS verification can send its lookups to a DNS-over-HTTPS provider, so a slow resolver costs a worker about 300 ms instead of up to ten seconds. Config merging now treats lists consistently: an empty list clears one, and separate configs passed to Firewall::create() replace lists as configs: includes do.

PHP 8.1–8.5 supported. The provider is opt-in. With no global.reverse_dns, verification uses PHP's own lookups, exactly as before, and nothing is sent to a third party. One merge change needs attention on upgrade if you pass more than one config to Firewall::create(); see Upgrading.

Added

DNS-over-HTTPS providers for reverse-DNS verification

PHP's gethostbyaddr() and dns_get_record() take no timeout. On a host without a local caching resolver, one slow nameserver held a worker for up to ten seconds before the circuit breaker could trip.

• global.reverse_dns.provider (#473, #475). Name a provider and every verifying rule uses it:

  global:
    reverse_dns:
      provider: cloudflare     # or google
      timeout_ms: 300          # per lookup; the default

  Each lookup is limited to timeout_ms, connecting and the TLS handshake included. The provider's address is pinned, so its own hostname is never looked up through the operating system's resolver, and the certificate is still checked against that hostname.

• Built-in providers cloudflare and google. They're available, but never selected by default. Naming one is the opt-in: from then on, the reverse-DNS names of visitors' addresses are sent to that provider, for requests that claim to be a crawler a rule verifies. Reverse DNS documents each provider: who runs it, what it receives, what it says it logs, and its terms.

• Per rule: verify_provider picks a different provider, and verify_timeout_ms a different limit.

• Your own provider or resolver. Define a provider under global.reverse_dns.providers (for a DNS-over-HTTPS resolver you run), or name a class implementing ReverseDnsResolverInterface with resolver and resolver_options. A resolver returns DNS data, never a verdict. The firewall still validates every hostname, checks the domain and makes the forward confirmation, so no resolver can make an allow rule match more than DNS confirms.

• Checked at startup. An unknown provider name, provider and resolver both set, or a resolver that can't be built stops the firewall starting. firewall-check --lint reports the same problems, and firewall-doctor says where each verifying rule's lookups go.

• Measured (#480, #482). 25 workers verifying one uncached address, with a provider that never answers: the first worker gives up at 306–308 ms, and the rest are refused in under a millisecond. Once a worker's connection is open, a verification costs 19–40 ms; the first costs 66–93 ms, TLS handshake included. The figures, and how to size timeout_ms and verify_claim_wait_ms for a provider, are in the docs.

• Checked weekly (#477, #484). A scheduled job runs each built-in provider through a reverse lookup, the forward lookup, a name with no record and a full verification, and opens an issue if one stops answering.

DnsOverHttpResolver needs PHP's curl extension. PHP's own lookups need nothing extra.

Safer verification, whatever makes the lookups

• At most two forward lookups per address. The owner of an address writes its PTR records, and could list any number of invented googlebot.com names to make every verification slow enough to trip the breaker for everyone.
• Verdicts and the breaker belong to the resolver that reached them. Switching to a provider starts afresh: a refusal from a lookup PHP couldn't finish isn't carried over for a day. A slow resolver only trips its own breaker.
• A lookup that couldn't say either way (a timeout, a server failure) is remembered for 60 seconds, not verify_negative_ttl, so one network blip doesn't refuse a real crawler for a day.

Fixed

• An included file couldn't clear a list (#474, #476). lockdown_allow: [] in an included file was ignored, so the earlier allowlist stayed in force. An empty list now replaces a list, as any other list does. [] over a map still changes nothing.
• Lists inside a legacy block:/bypass: entry merged position by position (#476). [b] over [a, c] gave [b, c]. They're now replaced whole. A legacy rule's config: still appends.
• Separate configs appended lists instead of replacing them (#481, #483). With Firewall::create([$base, $site]), a second trusted_proxies added to the first, so both were trusted, and lockdown_allow: [] couldn't clear the allowlist. Separate configs now merge lists as includes do. See Upgrading.

Upgrading

1. If you pass more than one config to Firewall::create(), a list set in two of them is now the later one's, not both joined together. That applies to trusted_proxies, lockdown_allow, a rule's verify_suffixes, and every other list. Root plugins: and a legacy rule's config: still append, and a later config's priority and enable: false still win. A host relying on two configs each adding to one list should move the whole list into one of them. A single-config site sees no difference.
2. Includes: an included file's [] now clears the list it overrides. Check any include that sets a list to [] meant as "nothing to add".
3. The compiled config cache reparses each configuration once after the upgrade, because its entry format changed. Nothing to do.
4. To use a provider, install PHP's curl extension (most hosts have it), set global.reverse_dns.provider, and read the provider's section of the Reverse DNS docs first. Set timeout_ms well above the provider's usual latency, since the limit includes the TLS handshake.

Full changelog: v2.37.0...v2.38.0