net/http: Content-Length is not set in outgoing request when using ioutil.NopCloser

[johansja]

What version of Go are you using (go version)?

$ go version
go version go1.13 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/1041775/Library/Caches/go-build"
GOENV="/Users/1041775/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/1041775/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/Cellar/go/1.13/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.13/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/1041775/projects/js-scripts/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/d6/809nhvwd23nd_wryrw60j6hdms016p/T/go-build703352534=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Start a httpbin server locally.

docker run -p 80:80 kennethreitz/httpbin

Run the following program

package main

import (
	"bytes"
	"io/ioutil"
	"log"
	"net/http"
)

func main() {
	reqBody := ioutil.NopCloser(bytes.NewBufferString(`{}`))
	req, err := http.NewRequest("POST", "http://localhost:80/post", reqBody)
	if err != nil {
		log.Fatalf("Cannot create request: %v", err)
	}
	res, err := http.DefaultClient.Do(req)
	if err != nil {
		log.Fatalf("Cannot do: %v", err)
	}
	defer res.Body.Close()
	resBody, err := ioutil.ReadAll(res.Body)
	if err != nil {
		log.Fatalf("Cannot read body: %v", err)
	}
	log.Printf("Response Body: %s", resBody)
}

What did you expect to see?

Content-Length header is set when it is received by the server.

What did you see instead?

Content-Length header is missing when it is received by the server.

2019/09/14 12:55:00 Response Body: {
  "args": {}, 
  "data": "{}", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept-Encoding": "gzip", 
    "Host": "localhost:80", 
    "Transfer-Encoding": "chunked", 
    "User-Agent": "Go-http-client/1.1"
  }, 
  "json": {}, 
  "origin": "172.17.0.1", 
  "url": "http://localhost:80/post"
}

Versus what I would receive if I use

reqBody := bytes.NewBufferString({})

.

2019/09/14 12:55:22 Response Body: {
  "args": {}, 
  "data": "{}", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept-Encoding": "gzip", 
    "Content-Length": "2", 
    "Host": "localhost:80", 
    "User-Agent": "Go-http-client/1.1"
  }, 
  "json": {}, 
  "origin": "172.17.0.1", 
  "url": "http://localhost:80/post"
}

[johansja]

I'm not sure if the Transport-Encoding chunked is making any differences here as well.

[odeke-em]

Hello there @johansja, thank you for filing this request and welcome to the Go project!

So, ioutil.NopCloser swallows its underlying reader and we can no longer get access to it directly as per even its signature https://golang.org/pkg/io/ioutil/#NopCloser and thus we can't guess its ContentLength currently, unfortunately! Therefore it turns into the case of streaming content that is continuously flowing in until it decides to send EOF on its own. that's why you see that Transport-Encoding: chunked which is used when streaming.

If you pass in reqBody as bytes.BufferString({}) you'll then see a Content-Length of 2, which we can inspect to give us a hint about the contentLength. There are many types of readers that you can use such as strings.Reader, bytes.Reader, *os.File etc but anyways the NopCloser makes the underlying reader unintrospectable.

I'm not sure if the Transport-Encoding chunked is making any differences here as well.

Transport-Encoding: chunked signifies that content is getting streamed because we don't know the lenght, so it is a symptom instead of a problem cause.

Here is a tip to make your test self contained and even speed up your development making it self contained and more fun, Go's net/http/httptest has a package we can use to create servers and echo our responses to us or even simpler to not have to spin up a server you can pass in an http.RoundTripper that mocks that echo backend and just dumps out the response for inspection and here is the code

With httptest.Server

https://play.golang.org/p/RghMmV5dhNKor inlined below:

package main

import (
	"bytes"
	"io/ioutil"
	"log"
	"net/http"
	"net/http/httptest"
	"net/http/httputil"
)

func main() {
	cst := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		reqBlob, _ := httputil.DumpRequest(r, true)
		w.Write(reqBlob)
	}))
	defer cst.Close()

	reqBody := ioutil.NopCloser(bytes.NewBufferString(`{}`))
	req, err := http.NewRequest("POST", cst.URL, reqBody)
	if err != nil {
		log.Fatalf("Cannot create request: %v", err)
	}
	res, err := http.DefaultClient.Do(req)
	if err != nil {
		log.Fatalf("Cannot do: %v", err)
	}
	defer res.Body.Close()
	resBody, err := ioutil.ReadAll(res.Body)
	if err != nil {
		log.Fatalf("Cannot read body: %v", err)
	}
	log.Printf("Response Body:\n%s", resBody)
}

With a custom roundtripper aka doesn't make an RPC

https://play.golang.org/p/aUCgAHYWbRS or inlined below:

package main

import (
	"bytes"
	"io/ioutil"
	"log"
	"net/http"
	"net/http/httputil"
)

type echoRoundTripper int

func (ert *echoRoundTripper) RoundTrip(req *http.Request) (*http.Response, error) {
	reqBlob, _ := httputil.DumpRequestOut(req, true)
	res := &http.Response{
		Status:     "200 OK",
		StatusCode: 200,
		Body:       ioutil.NopCloser(bytes.NewReader(reqBlob)),
	}

	return res, nil
}

func main() {
	reqBody := ioutil.NopCloser(bytes.NewBufferString(`{}`))
	req, err := http.NewRequest("POST", "http://localhost:80/post", reqBody)
	if err != nil {
		log.Fatalf("Cannot create request: %v", err)
	}

	client := &http.Client{Transport: new(echoRoundTripper)}
	res, err := client.Do(req)
	if err != nil {
		log.Fatalf("Cannot do: %v", err)
	}
	defer res.Body.Close()
	resBody, err := ioutil.ReadAll(res.Body)
	if err != nil {
		log.Fatalf("Cannot read body: %v", err)
	}
	log.Printf("Response Body:\n%s", resBody)
}

I shall close this issue as working as intended, but perhaps we could examine documenting the cases for which ContentLength cannot be inferred directly.

** disclaimer: if we come to a consensus, we can try hard to inspect the ioutil.NopCloser by an unsafe cast, and then with that reader's length if inspectable but that might violate user expectations and previous older programs, and also creates brittle code and importing unsafe in net/http might be frowned on **

[odeke-em]

In the docs for net/http.NewRequestWithContext https://golang.org/pkg/net/http/#NewRequestWithContext which just got added in Go1.13

and by following net/http.NewRequest https://golang.org/pkg/net/http/#NewRequest which links back to NewRequestWithContext, we already document this so nothing left to do here.

[johansja]

Thanks for looking into this, @odeke-em . The explanation is giving me more knowledge on Go as well.