Skip to content

doh: pooled connections outlive resolver idle-timeouts — recurring "unexpected EOF" / 502 bursts (Quad9 kills idle conns at ≤30s vs 3m IdleConnTimeout) #241

Description

@ShiroiKuma0

Summary

intra/doh/doh.go h2() configures the DoH http.Transport with IdleConnTimeout: 3 * time.Minute, ForceAttemptHTTP2: true, and no HTTP/2 keep-alive health-check (ReadIdleTimeout unset). Several popular resolvers close idle DoH connections far sooner than 3 minutes. Measured from the same network, same day (raw TLS connection reused after an idle period):

resolver idle time survived
dns.quad9.net EOF at ≤ 30 s
dns.mullvad.net EOF at ≤ 90 s
cloudflare-dns.com alive at > 240 s

With Quad9 (or any resolver whose idle timeout is under 3 minutes), every DNS lull longer than the server's idle window leaves the entire pool populated with dead connections. The next queries are written into them, hang ~8–10 s, and surface as Post "https://dns.quad9.net/dns-query": unexpected EOF or http-status: 502. The maxEOFTries (2) retries often just pick the next dead pooled connection, so bursts of parallel queries (A+AAAA × several apps) fail together. Fresh dials then succeed — until the next lull. The result is a chronic cycle of 10–30 s DNS outages every few minutes.

Downstream effects observed on-device (Android, v055x tag, fd3dbcd): alg mappings expire without refresh (alg: dns64: … realips([]) … until: 0s) so established flows of allowed apps die; OS connectivity probes routed through the tunnel fail, so Android network validation flaps device-wide (even VPN-excluded apps feel it). Log signature:

retrier: read: Base: [<laddr><=9.9.9.9:443]; (rtt: 3.02s / …); b: 0/576 (tee: 0); err: EOF
dns: udp: for <uid> err! tot: 0, t: 10.3s, Post "https://dns.quad9.net/dns-query": unexpected EOF
dns: udp: for <uid> err! tot: 0, t: 3.4s, http-status: 502

Verified it is NOT the network or the resolver being down: from a host on the same LAN, fresh requests to dns.quad9.net answered HTTP 200 in ~120 ms throughout, while the device failed continuously. Only reuse-after-idle is poisoned.

Fix (tested on-device)

--- a/intra/doh/doh.go
+++ b/intra/doh/doh.go
@@ func h2(d protect.DialFn, c *tls.Config) *http.Transport {
-	return &http.Transport{
-		DialContext:         asDialContext(d),
-		ForceAttemptHTTP2:   true,
-		IdleConnTimeout:     3 * time.Minute,
+	tr := &http.Transport{
+		DialContext:       asDialContext(d),
+		ForceAttemptHTTP2: true,
+		// keep pooled conns shorter-lived than any observed server-side idle timeout
+		IdleConnTimeout:     10 * time.Second,
 		TLSHandshakeTimeout: 7 * time.Second,
 		// Android's DNS-over-TLS sets it to 30s
 		ResponseHeaderTimeout: 20 * time.Second,
 		// SNI (hostname) must always be inferred from http-request
 		TLSClientConfig: c,
 	}
+	// health-check h2 conns with PING so half-dead ones are evicted
+	// instead of eating a query first (net/http never does this by default)
+	if t2, err := http2.ConfigureTransports(tr); err == nil && t2 != nil {
+		t2.ReadIdleTimeout = 10 * time.Second
+		t2.PingTimeout = 5 * time.Second
+	}
+	return tr
 }

golang.org/x/net is already a module dependency, so the only new import is golang.org/x/net/http2.

Validation

Same device, same resolver (Quad9), worst-case traffic: bursts of 10 parallel uncached queries after guaranteed > 30 s idle gaps, ~40 cycles over 33 minutes. Stock: query-failure bursts on nearly every cycle. Patched: zero idle-pool failures; the log instead shows the health-check discovering and evicting the server-closed connections (retrier: read … err: EOF with no query error following), roughly one per idle cycle.

One caveat worth noting: with the h2 PING enabled we observed a single rare incident of stream error: stream ID N; PROTOCOL_ERROR resets taking out one burst on one connection (once in ~40 cycles; recovered on redial). If that turns out to be an interaction between the h2 health-check and the custom dialer, the IdleConnTimeout reduction alone — which already removes the corpse-pool state structurally — may be the safer minimal fix.

Happy to send this as a PR.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions