net/http: Server removes all explicitly set `Transfer-Encoding` headers

[seoester]

The net/http server removes all explicitly set Transfer-Encoding headers (when operating in HTTP/1.1 mode).

This bug was introduced by the fix of #15960 (f9b4556), sorry for any ambiguous wording on my end.

Example:

package main

import (
    "compress/gzip"
    "net/http"
    "strings"
)

const Level int = -1

func myHandler(w http.ResponseWriter, r *http.Request) {
    if strings.Contains(r.Header.Get("TE"), "gzip") {
        writer, err := gzip.NewWriterLevel(w, Level)
        if err != nil {
            panic(err)
        }
        w.Header().Set("Transfer-Encoding", "gzip")
        writer.Write([]byte("Some content..."))
    } else {
        w.Write([]byte("Couldn't gzip (not accepted)"))
    }
}

func main() {
    http.ListenAndServe(":8080", http.HandlerFunc(myHandler))
}

For clients accepting a gzip transfer-coding, net/http removes the explicitly set gzip from the Transfer-Encoding header and only sends chunked.

To fix both issues there must be a check for the header field's value (== "chunked").
E.g. (the line introduced by #15960 also has to be removed):

        ...
        if hasTE && te == "identity" {
            cw.chunking = false
            w.closeAfterReply = true
        } else if hasTE && te == "chunked" {
            cw.chunking = true
            setHeader.transferEncoding = "chunked"
            delHeader("Transfer-Encoding")
        } else {
            // HTTP/1.1 or greater: use chunked transfer encoding
            ...

To reiterate:

• An explicitly set transfer-coding of chunked should not result in a duplicate Transfer-Encoding: chunked header
• Whenever another transfer-coding is set, chunked must be applied also, but the explicitly set transfer-encoding should be kept in the header

Details can be found in RFC 2616 Section 3.6.

[ianlancetaylor]

What is our best move for 1.7 here? I want us to be very careful to avoid the kind of problem we had with 1.6, where late changes broke HTTP2 support. Should we roll back the fix for #15960 and leave all of this for 1.8?

[seoester]

Pre #15960 is only non-rfc-compliant in one edge case, any client I tested (chrom{e, ium}, firefox, wget) didn't break however.

That being said though I personally think the suggested fix (+ removing the line introduced by #15960) is rather safe from side effects.

[adg]

I sent a change. I think it is a low-risk change, as the net change now only affects users that explicitly set Transfer-Encoding: chunked in their handlers. We also have tests now. I think this is preferable to a rollback.

[gopherbot]

CL https://golang.org/cl/24130 mentions this issue.