Validator probably shouldn't send Accept-Encoding headers

[schoen]

Per https://community.letsencrypt.org/t/unable-to-get-certificate-error-reading-http-response-body-unexpected-eof/63987/, there are some configurations with buggy implementations of gzip or deflate encoding in HTTP replies. (This seems to involve reverse proxies getting confused about the semantics of Content-Length: headers, although I haven't entirely chased it down.)

While this is definitely a standards-conformance problem on the other implementations' end, and not a bug in Boulder's validation, there is no real benefit to using gzip encoding for sending HTTP-01 challenge tokens because these tokens are intentionally random; compressing random data, by the pigeonhole principle, should usually make it larger rather than smaller.

So, it doesn't seem like there would be any loss from simply not having Boulder send Accept-Encoding: , and dropping it might avoid an entire area of bugs and perhaps even attack surface on the Boulder side.

[alexzorin]

I've occasionally seen servers that send a Content-Encoding: gzip response even in the absence of Accept-Encoding. This often happens with caching Varnish servers that do not Vary: Accept-Encoding properly.

I mention this because setting http.Transport.DisableCompression: true will result in the the HTTP client not decoding the unsolicited gzip response.

Removing the request header sounds fine, but attention should be paid to the response headers too, so the change doesn't break a different kind of misconfiguration that may be functional today.

[jsha]

This hasn't turned out to be a big issue in the last couple of years, and as @alexzorin points out, there are many servers on the Internet that unconditionally gzip responses, so the risk of breakage if we were to turn this on would be significant.

Closing for now. As always, willing to reconsider with new information.