Skip to content

v1.7.4

Choose a tag to compare

@semihalev semihalev released this 09 Aug 16:06
· 55 commits to main since this release
e597a1c

A resilience release: request-level work budgets, standards-based negative caching, and the fixes from a production outage post-mortem — plus a measurably faster hot path. Recommended for all deployments; open resolvers exposed to untrusted query load benefit most.

Recursion firewall (#527)

  • Request-tree work budgets. Outbound transport attempts, resolver-internal child queries, and DNSSEC operations (signature verifications, DS digests, NSEC3 hashes, per-RRset/candidate fan-out, process-wide crypto concurrency) are accounted against one ledger spanning the complete request tree — retries, UDP→TCP fallbacks, and nested DS/DNSKEY/NS lookups included — so one hostile query cannot amplify into unbounded upstream traffic or crypto work.
  • Three modes, shadow by default. off / shadow / enforce via [recursion_firewall] mode. Shadow records budget crossings in metrics plus a rate-limited log line naming the query, without changing any response; enforce terminates over-budget trees with SERVFAIL and an RFC 8914 Extended DNS Error. Calibrate from the dnssec_work_per_request and dns_recursion_fanout_ratio histograms before enforcing.
  • RFC 9520 §3.1 retry ceiling. At most three attempts per (question, server, transport) tuple per resolution, always on.

Negative caching (#527)

  • RFC 9520 resolution-failure caching. SERVFAILs and failed-authority state live in a dedicated bounded cache with exponential backoff (5s→5m) and zone-wide reachability entries; after backoff expiry, concurrent retries elect a single upstream probe instead of stampeding. Request-local failures (budget rejections, attempt limits, cancellation) are never admitted to shared state. Cached failures answer with EDE 13. Kill switch: rfc9520. First field hour on a public node: ~16 answers/s served from failure state — retry load that no longer reaches upstreams.
  • RFC 8020 NXDOMAIN subtree cuts and RFC 8198 aggressive NSEC/NSEC3 synthesis. Locally validated denial proofs answer later negative queries without upstream traffic. Admission requires this resolver's own validation — the AD bit is never trusted — with exact-response provenance and fingerprint sealing; NSEC3 Opt-Out is excluded by design. The proof indexes carry their own per-zone and global entry/byte bounds so hostile denial churn cannot evict ordinary answers. Kill switch: rfc8198. (~4.8k upstream lookups/hour avoided on the same node.)

Outage containment (#527)

Root-caused from a 2026-07-28 production incident (1.43M goroutines, 93% blocked behind cache segment locks during a partial upstream outage):

  • Proportional cache eviction. Segment-clearing eviction (dropping 10–60% of entries in bulk while writers queued uncancellably) is replaced by self-paying eviction: an over-capacity insert evicts at most two entries under the lock it already holds. Worst single-insert loss drops from 50,003 entries to 1; the eviction-heavy insert benchmark improves 378→94 ns/op.
  • Ghost-entry fix. Backward-shift deletion stopped at the first unmovable entry, stranding later cluster entries — present and counted but unfindable until an incidental grow. Rewritten to the canonical linear-probing delete (Knuth 6.4R), which also removes an O(cluster²) cost from every delete.
  • In-flight ceilings and per-zone fairness. A hard cap on concurrent zone-level lookups plus a per-zone quota (the analog of BIND's fetches-per-zone): a popular destination going dark sheds itself with a scoped EDE while every other zone keeps resolving at full speed, through every phase of an incident. Detached IPv6 NS enrichment draws from a bounded pool instead of growing at arrival-rate × timeout. Sheds are observable via dns_resolution_shed_total{scope}.

DNSSEC hardening (#527)

  • Opt-Out denial proofs no longer set AD (RFC 5155 §9.2). The proof still validates; the response is correctly marked insecure instead.
  • NSEC3 evaluation on a prepared ring. Each response's NSEC3 set is bound to the RRSIG signer zone and one parameter tuple, then evaluated as a sorted ring — removing the per-record hash multiplier an attacker-authored zone could exploit. Responses mixing NSEC3 parameter tuples are rejected fail-closed per RFC 5155 §7.2/§8.2.
  • Sharper validation plumbing. Canonical wire-label handling in wildcard next-closer derivation (Unicode case folding could alias distinct octet names), deterministic deduplicated candidate ordering in DS/RRSIG verification, and RFC 5011 trust-anchor refresh brought under the same work accounting.

Performance (#527)

  • End-to-end request deadline, kept off the hot path. querytimeout now bounds the whole pipeline from ingress on every transport (UDP/TCP/DoT/DoH/DoQ). A lazily-armed deadline context keeps its cost invisible: versus a standard context.WithTimeout at ingress, pipeline overhead drops 481→200 ns (7→1 allocs) and a positive cache hit 760→471 ns (15→9 allocs).
  • Cold-cache parity with PDNS Recursor 5.4.1 over 50k live queries: 641 vs 627 qps, 0 lost (PDNS: 14), response-code distribution identical to within 0.05% — with every new accounting and validation layer active.

Operator notes

  • querytimeout is now end-to-end: cache dedup waits, DNS64 subqueries, and failover all count against it, and fallbackservers are not tried once a blackholed upstream has consumed the window.
  • The legacy negative answer cache is retired: the full cachesize now backs positive/NXDOMAIN entries, and SERVFAILs use the RFC 9520 failure cache (failure_cache_* settings under [recursion_firewall]).
  • The config schema is now v1.7.4 (rfc8198, rfc9520, [recursion_firewall]). Older config files get the out-of-version notice — regenerate to see the new sections, or keep running: omitted settings mean shadow mode with both switches on.
  • The per-zone in-flight quota defaults to maxconcurrentqueries/16 (min 16); very-high-QPS cold-cache deployments should size maxconcurrentqueries accordingly.

Build

  • Dependencies: quic-go 0.61.0 (#523), prometheus/client_golang 1.24.1 (#522), k8s.io 0.36.3 (#521), CI action bumps (#524, #526).

Full changelog: v1.7.3...v1.7.4