Skip to content

net/http: Transport doesn't support specify Transfer-Encoding explicitly #33733

Closed
@zema1

Description

@zema1

What version of Go are you using (go version)?

$ go version
go version go1.12.6 darwin/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/zg/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/zg/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/zg/Development/gunkit/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/lg/cht6v2795qn9z07kmcl6h3c40000gn/T/go-build863622379=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

I know http.Transport will use chunked Encoding automatically when Request has body and Request.Content-Length was set < 0. But this troubled me when I use net.http in my own penetration testing tool, which always need to build a custom chunked body. The code simply looks like the following snippet:

buf := &bytes.Buffer{}
ck := NewCustomChunkedWriter(buf)
ck.WriteString("test")
ck.Close()
req, _ = http.NewRequest("POST", "http://xx.com", buf)
req.Header.Set("Transfer-Encoding", "chunked")

Howover, it didn't work as expected. I tried to trace the roundtrip method and found that Transfer-Encoding is in a map named reqWriteExcludeHeader, which looks like a whitelist when writing request header.

And if I set req.Transfer-Encoding = []string{"chunked"}, the request body will be chunked twice! one from my customChunkedWriter, the other from the internal chunked writer.

Finally, I believe there is actually no way to make it. I think It's not a intended behavior. It should be great if it acts like Accept-Encoding.

If the Transport requests gzip on its own and gets a gzipped response, it's transparently decoded in the Response.Body. However, if the user explicitly requested gzip it is not automatically uncompressed.

Related issue: #28026

What did you expect to see?

If I send Transfer-Encoding explicitly, transport should keep the header and do nothing about the request body.

What did you see instead?

Described as above.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions