Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http: Server sets both Content-Length & Transfer-Encoding: chunked #9987

Closed
tedjp opened this issue Feb 24, 2015 · 1 comment

Comments

Projects
None yet
3 participants
@tedjp
Copy link

commented Feb 24, 2015

When implementing Tranfer-Encoding (transfer-coding, TE) other than chunked in an http.Server, Go produces an invalid combination of HTTP response headers — it provides both Content-Length and Transfer-Encoding. It should not be writing a Content-Length header.

RFC 2616 Section 4.4:

If a Content-Length header field (section 14.13) is present, its decimal value in OCTETs represents both the entity-length and the transfer-length. The Content-Length header field MUST NOT be sent if these two lengths are different (i.e., if a Transfer-Encoding header field is present). If a message is received with both a Transfer-Encoding header field and a Content-Length header field, the latter MUST be ignored.

I am using go version go1.4.1 linux/amd64.

To reproduce the problem, compile and run an HTTP server that implements gzip Transfer-Encoding:

package main

import "bytes"
import "compress/gzip"
import "log"
import "net/http"
import "strings"

func gz(input []byte) []byte {
    var buf bytes.Buffer
    w := gzip.NewWriter(&buf)
    w.Write(input)
    w.Close()
    return buf.Bytes()
}

var response = []byte("Hello, world!\n")

func main() {
    http.HandleFunc("/", func(w http.ResponseWriter, req *http.Request) {
        // This is not strictly correct, but close enough for demoing
        if strings.Contains(strings.ToLower(req.Header.Get("TE")), "gzip") {
            // Go automatically wraps this in chunked as required by spec
            w.Header().Set("Transfer-Encoding", "gzip")
            // Uncomment this to print an error, but produce a valid response
            //w.Header().Set("Content-Length", "-1")
            w.Write(gz(response))
        } else {
            w.Write(response)
        }
    })

    log.Fatal(http.ListenAndServe(":8080", nil))
}

Then use curl to query the server. curl is one of the few clients that implement Transfer-Encoding, and it knows how to handle the Transfer-Encoding header in response to this request with the TE request header.

$ curl -i -H 'TE: gzip' http://localhost:8080/
HTTP/1.1 200 OK
Transfer-Encoding: gzip
Date: Tue, 24 Feb 2015 20:51:18 GMT
Content-Length: 38
Content-Type: application/x-gzip
Transfer-Encoding: chunked

Hello, world!

http.Server logs an error saying that the Content-Length was set, but it was actually calculated in http.Server's chunkWriter.writeHeader().

http: WriteHeader called with both Transfer-Encoding of "gzip" and a Content-Length of 38

Setting the Content-Length to an invalid value (such as -1, by uncommenting line 26 of the source), produces correct output, but also logs an error message.

http: invalid Content-Length of "-1"

What did you expect to see?

A Transfer-Encoding header (either "gzip, chunked" or two separate headers, "gzip" and "chunked") and no Content-Length header.

What did you see instead?

Both a Content-Length & Transfer-Encoding header.

The problem appears to be where the contentLength is calculated in src/net/http/server.go near line 795:

if w.handlerDone && !trailers && bodyAllowedForStatus(w.status) && header.get("Content-Length") == "" && (!isHEAD || len(p) > 0) {
    w.contentLength = int64(len(p))
    setHeader.contentLength = strconv.AppendInt(cw.res.clenBuf[:0], int64(len(p)), 10)
}

This is causing the Content-Length header to be set.

Then around line 860 the presence of the Transfer-Encoding header is determined:

te := header.get("Transfer-Encoding")
hasTE := te != ""
if hasCL && hasTE && te != "identity" {
    // TODO: return an error if WriteHeader gets a return parameter
    // For now just ignore the Content-Length.
    w.conn.server.logf("http: WriteHeader called with both Transfer-Encoding of %q and a Content-Length of %d",            te, w.contentLength)
    delHeader("Content-Length")
    hasCL = false
}

But I think the fact that w.contentLength is still set to something other than -1 means it is unaffected by the presence of "Content-Length" in excludeHeader (via delHeader()) and the Content-Length is included in the output headers.

A couple of options that probably fix this:

  1. Move the calculation of the Content-Length & the definition of hasCL to occur after hasTE is defined, and only if !hasTE. (Neither the calculated content length nor hasCL is used before the hasCL && hasTE condition.); or
  2. Add && header.get("Transfer-Encoding") == "" to the condtion near line 795 to prevent the Content-Length header being produced.

(In my real world implementation I set the Content-Type explicitly, so I'm not concerned about the Content-Type being guessed incorrectly as it is in this example, where it should be "text/plain". Maybe it shouldn't be detected if the user set a Transfer-Encoding?)

@bradfitz bradfitz added this to the Go1.5 milestone Feb 24, 2015

@bradfitz bradfitz self-assigned this Feb 24, 2015

@bradfitz bradfitz changed the title http.Server sets both Content-Length & Transfer-Encoding: chunked net/http: Server sets both Content-Length & Transfer-Encoding: chunked Feb 24, 2015

@gopherbot

This comment has been minimized.

Copy link

commented May 4, 2015

CL https://golang.org/cl/9638 mentions this issue.

@bradfitz bradfitz closed this in c661cb0 May 6, 2015

@golang golang locked and limited conversation to collaborators Jun 25, 2016

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.