Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http: Content-Length is not set in outgoing request when using ioutil.NopCloser #34295

Closed
johansja opened this issue Sep 14, 2019 · 4 comments
Closed
Labels

Comments

@johansja
Copy link

@johansja johansja commented Sep 14, 2019

What version of Go are you using (go version)?

$ go version
go version go1.13 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/1041775/Library/Caches/go-build"
GOENV="/Users/1041775/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/1041775/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/Cellar/go/1.13/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.13/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/1041775/projects/js-scripts/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/d6/809nhvwd23nd_wryrw60j6hdms016p/T/go-build703352534=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Start a httpbin server locally.

docker run -p 80:80 kennethreitz/httpbin

Run the following program

package main

import (
	"bytes"
	"io/ioutil"
	"log"
	"net/http"
)

func main() {
	reqBody := ioutil.NopCloser(bytes.NewBufferString(`{}`))
	req, err := http.NewRequest("POST", "http://localhost:80/post", reqBody)
	if err != nil {
		log.Fatalf("Cannot create request: %v", err)
	}
	res, err := http.DefaultClient.Do(req)
	if err != nil {
		log.Fatalf("Cannot do: %v", err)
	}
	defer res.Body.Close()
	resBody, err := ioutil.ReadAll(res.Body)
	if err != nil {
		log.Fatalf("Cannot read body: %v", err)
	}
	log.Printf("Response Body: %s", resBody)
}

What did you expect to see?

Content-Length header is set when it is received by the server.

What did you see instead?

Content-Length header is missing when it is received by the server.

2019/09/14 12:55:00 Response Body: {
  "args": {}, 
  "data": "{}", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept-Encoding": "gzip", 
    "Host": "localhost:80", 
    "Transfer-Encoding": "chunked", 
    "User-Agent": "Go-http-client/1.1"
  }, 
  "json": {}, 
  "origin": "172.17.0.1", 
  "url": "http://localhost:80/post"
}

Versus what I would receive if I use

reqBody := bytes.NewBufferString({})
.

2019/09/14 12:55:22 Response Body: {
  "args": {}, 
  "data": "{}", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept-Encoding": "gzip", 
    "Content-Length": "2", 
    "Host": "localhost:80", 
    "User-Agent": "Go-http-client/1.1"
  }, 
  "json": {}, 
  "origin": "172.17.0.1", 
  "url": "http://localhost:80/post"
}
@johansja

This comment has been minimized.

Copy link
Author

@johansja johansja commented Sep 14, 2019

I'm not sure if the Transport-Encoding chunked is making any differences here as well.

@odeke-em

This comment has been minimized.

Copy link
Member

@odeke-em odeke-em commented Sep 14, 2019

Hello there @johansja, thank you for filing this request and welcome to the Go project!

So, ioutil.NopCloser swallows its underlying reader and we can no longer get access to it directly as per even its signature https://golang.org/pkg/io/ioutil/#NopCloser and thus we can't guess its ContentLength currently, unfortunately! Therefore it turns into the case of streaming content that is continuously flowing in until it decides to send EOF on its own. that's why you see that Transport-Encoding: chunked which is used when streaming.

If you pass in reqBody as bytes.BufferString({}) you'll then see a Content-Length of 2, which we can inspect to give us a hint about the contentLength. There are many types of readers that you can use such as strings.Reader, bytes.Reader, *os.File etc but anyways the NopCloser makes the underlying reader unintrospectable.

I'm not sure if the Transport-Encoding chunked is making any differences here as well.

Transport-Encoding: chunked signifies that content is getting streamed because we don't know the lenght, so it is a symptom instead of a problem cause.

Here is a tip to make your test self contained and even speed up your development making it self contained and more fun, Go's net/http/httptest has a package we can use to create servers and echo our responses to us or even simpler to not have to spin up a server you can pass in an http.RoundTripper that mocks that echo backend and just dumps out the response for inspection and here is the code

With httptest.Server

https://play.golang.org/p/RghMmV5dhNKor inlined below:

package main

import (
	"bytes"
	"io/ioutil"
	"log"
	"net/http"
	"net/http/httptest"
	"net/http/httputil"
)

func main() {
	cst := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		reqBlob, _ := httputil.DumpRequest(r, true)
		w.Write(reqBlob)
	}))
	defer cst.Close()

	reqBody := ioutil.NopCloser(bytes.NewBufferString(`{}`))
	req, err := http.NewRequest("POST", cst.URL, reqBody)
	if err != nil {
		log.Fatalf("Cannot create request: %v", err)
	}
	res, err := http.DefaultClient.Do(req)
	if err != nil {
		log.Fatalf("Cannot do: %v", err)
	}
	defer res.Body.Close()
	resBody, err := ioutil.ReadAll(res.Body)
	if err != nil {
		log.Fatalf("Cannot read body: %v", err)
	}
	log.Printf("Response Body:\n%s", resBody)
}

With a custom roundtripper aka doesn't make an RPC

https://play.golang.org/p/aUCgAHYWbRS or inlined below:

package main

import (
	"bytes"
	"io/ioutil"
	"log"
	"net/http"
	"net/http/httputil"
)

type echoRoundTripper int

func (ert *echoRoundTripper) RoundTrip(req *http.Request) (*http.Response, error) {
	reqBlob, _ := httputil.DumpRequestOut(req, true)
	res := &http.Response{
		Status:     "200 OK",
		StatusCode: 200,
		Body:       ioutil.NopCloser(bytes.NewReader(reqBlob)),
	}

	return res, nil
}

func main() {
	reqBody := ioutil.NopCloser(bytes.NewBufferString(`{}`))
	req, err := http.NewRequest("POST", "http://localhost:80/post", reqBody)
	if err != nil {
		log.Fatalf("Cannot create request: %v", err)
	}

	client := &http.Client{Transport: new(echoRoundTripper)}
	res, err := client.Do(req)
	if err != nil {
		log.Fatalf("Cannot do: %v", err)
	}
	defer res.Body.Close()
	resBody, err := ioutil.ReadAll(res.Body)
	if err != nil {
		log.Fatalf("Cannot read body: %v", err)
	}
	log.Printf("Response Body:\n%s", resBody)
}

I shall close this issue as working as intended, but perhaps we could examine documenting the cases for which ContentLength cannot be inferred directly.

** disclaimer: if we come to a consensus, we can try hard to inspect the ioutil.NopCloser by an unsafe cast, and then with that reader's length if inspectable but that might violate user expectations and previous older programs, and also creates brittle code and importing unsafe in net/http might be frowned on **

@odeke-em odeke-em changed the title Content-Length is not set in outgoing HTTP request when using ioutil.NopCloser net/http: Content-Length is not set in outgoing request when using ioutil.NopCloser Sep 14, 2019
@odeke-em

This comment has been minimized.

Copy link
Member

@odeke-em odeke-em commented Sep 15, 2019

In the docs for net/http.NewRequestWithContext https://golang.org/pkg/net/http/#NewRequestWithContext which just got added in Go1.13
Screen Shot 2019-09-14 at 6 57 26 PM

and by following net/http.NewRequest https://golang.org/pkg/net/http/#NewRequest which links back to NewRequestWithContext, we already document this so nothing left to do here.

@odeke-em odeke-em closed this Sep 15, 2019
@johansja

This comment has been minimized.

Copy link
Author

@johansja johansja commented Sep 15, 2019

Thanks for looking into this, @odeke-em . The explanation is giving me more knowledge on Go as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.