proposal: net/http: accept "x-gzip" content encoding value

[melardev]

Hi, would it be good if I submit the following diff file?
It basically treats x-gzip as gzip, as noted in the HTTP 0.9 RFC: https://www.ietf.org/rfc/rfc1945.txt section 3.5
For the request header, that may not be that important, but for the response it may save more than one response from legacy servers.

@@ -2012,7 +2012,8 @@ func (pc *persistConn) readLoop() {
 		}
 
 		resp.Body = body
-		if rc.addedGzip && strings.EqualFold(resp.Header.Get("Content-Encoding"), "gzip") {
+		if rc.addedGzip && (strings.EqualFold(resp.Header.Get("Content-Encoding"), "gzip") ||
+			strings.EqualFold(resp.Header.Get("Content-Encoding"), "x-gzip")) {
 			resp.Body = &gzipReader{body: body}
 			resp.Header.Del("Content-Encoding")
 			resp.Header.Del("Content-Length")
@@ -2368,9 +2369,9 @@ func (pc *persistConn) roundTrip(req *transportRequest) (resp *Response, err err
 	// requested it.
 	requestedGzip := false
 	if !pc.t.DisableCompression &&
-		req.Header.Get("Accept-Encoding") == "" &&
 		req.Header.Get("Range") == "" &&
 		req.Method != "HEAD" {
+
 		// Request gzip only, not deflate. Deflate is ambiguous and
 		// not as universally supported anyway.
 		// See: https://zlib.net/zlib_faq.html#faq39
@@ -2383,8 +2384,14 @@ func (pc *persistConn) roundTrip(req *transportRequest) (resp *Response, err err
 		// We don't request gzip if the request is for a range, since
 		// auto-decoding a portion of a gzipped document will just fail
 		// anyway. See https://golang.org/issue/8923
-		requestedGzip = true
-		req.extraHeaders().Set("Accept-Encoding", "gzip")
+		if req.Header.Get("Accept-Encoding") == "" || req.Header.Get("Accept-Encoding") == "gzip" {
+			requestedGzip = true
+			req.extraHeaders().Set("Accept-Encoding", "gzip")
+		} else if req.Header.Get("Accept-Encoding") == "x-gzip" {
+			requestedGzip = true
+			req.extraHeaders().Set("Accept-Encoding", "x-gzip")
+		}
+
 	}
 
 	var continueCh chan struct{}

[andybons]

@neild @bradfitz

[andybons]

@melardev are you experiencing this in the wild? Which servers still serve x-gzip as their content-encoding?

[melardev]

@andybons actually not, but I was trying to write some HTTP client code for HTTP 0.9 and reviewing the RFC this showed up, I asked myself if the remote server returns such response, would be golang able to decode it or not, looking at the code it seems there may be an issue if that code path is triggered, so I wanted to ask here if it would be interesting to add support for it and see what you think about that.

[melardev]

Also, I don't know exactly when that function is being triggered, by default using HttpClient.Do() it is not, it would be interesting to know what would have happened if the client already adds the header "Accept-Encoding" to "gzip", the code as of now, it would not set the requestedGzip boolean flag to true, look at:
https://github.com/golang/go/blob/master/src/net/http/transport.go#L2504
It only sets the requestedGzip boolean to true if the header is absent and not when the header is present with gzip. But again, don't know exactly when that code is triggered.

[melardev]

@andybons Digging a little bit further I detected a bug indeed, this same code is at h2_bundle.go which has a copy paste code (as the comment also state):
https://github.com/golang/go/blob/master/src/net/http/h2_bundle.go#L7500
Then the following check is bypassed because requestedGzip is false.
https://github.com/golang/go/blob/master/src/net/http/h2_bundle.go#L8474
So to reproduce the bug run the following:

client := http.Client{}
	request, err := http.NewRequest("GET", "https://httpbin.org/gzip", nil)
	request.Header.Set("Accept-Encoding", "gzip")

	if err != nil {
		log.Fatalln(err)
	}

	resp, err := client.Do(request)
	if err != nil {
		log.Fatalln(err)
	}

	content, _ := ioutil.ReadAll(resp.Body)
        // Garbage will be printed, gzip is not decoded!
	log.Println(string(content))

If we remove the line

request.Header.Set("Accept-Encoding", "gzip")

The code would work fine.
This would be my first contribution to such a big prorject, can I make a pull request to fix this bug?

[rsc]

We are not trying to support HTTP/0.9 - we removed support for it.
Nor are we trying to support hypothetical uses.
If there's no actual x-gzip use in the wild, let's not add the complexity here.
Do you know of any important need for x-gzip in real-world use cases?

[melardev]

@rsc Hi, no I don't have a real-word use case, I just read that reviewing the HTTP 0.9 spec, I think we can close the issue then.

[rsc]

Based on the discussion above, this seems like a likely decline.

[rsc]

No change in consensus, so declined.