Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http: XML ContentLength for GET request differs between clients #19138

Closed
smasher164 opened this issue Feb 17, 2017 · 3 comments

Comments

Projects
None yet
3 participants
@smasher164
Copy link
Contributor

commented Feb 17, 2017

Please answer these questions before submitting your issue. Thanks!

What did you do?

Reading the ContentLength for an XML Response always returns -1. I'm not sure if this is intended behavior, since both curl and Chrome return the proper values. Changing the user-agent and cookie jar doesn't seem to affect the output, but sending a HEAD instead of GET returns the right value. Running this example outputs

$ go run main.go
-1
-1
-1
179
2596
1021

What did you expect to see?

$ go run main.go
179
2596
1021
179
2596
1021

FWIW, I saw the same behavior in 1.7.4 as well.

System details

go version go1.8 darwin/amd64
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/akhil/work"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/85/n35jf3zx5gl1y19r6thls_8m0000gn/T/go-build355476889=/tmp/go-build -gno-record-gcc-switches -fno-common"
CXX="clang++"
CGO_ENABLED="1"
PKG_CONFIG="pkg-config"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
GOROOT/bin/go version: go version go1.8 darwin/amd64
GOROOT/bin/go tool compile -V: compile version go1.8 X:framepointer
uname -v: Darwin Kernel Version 15.6.0: Thu Jun 23 18:25:34 PDT 2016; root:xnu-3248.60.10~1/RELEASE_X86_64
ProductName:	Mac OS X
ProductVersion:	10.11.6
BuildVersion:	15G31
lldb --version: lldb-360.1.70
@kevinburke

This comment has been minimized.

Copy link
Contributor

commented Feb 17, 2017

I modified your example a little bit so it behaves the way you expect. rest.DefaultTransport just dumps the raw request to stderr.

package main

import (
	"fmt"
	"log"
	"net/http"

	"github.com/kevinburke/rest"
)

func main() {
	examples := []string{
		"http://www.xmlfiles.com/examples/note.xml",
		//"https://www.gpo.gov/fdsys/bulkdata/BILLS/115/1/hconres/BILLS-115hconres18rds.xml",
		//"https://www.w3schools.com/xml/cd_catalog.xml",
	}
	http.DefaultClient.Transport = &rest.Transport{Debug: true, Output: os.Stderr}
	for i := range examples {
		req, err := http.NewRequest("GET", examples[i], nil)
		if err != nil {
			log.Fatalln(err)
		}
		req.Header.Set("Accept-Encoding", "identity")
		resp, err := http.DefaultClient.Do(req)
		if err != nil {
			log.Fatalln(err)
		}
		fmt.Println(resp.ContentLength)
		resp.Body.Close()
	}

	for i := range examples {
		resp, err := http.Head(examples[i])
		if err != nil {
			log.Fatalln(err)
		}
		fmt.Println(resp.ContentLength)
		resp.Body.Close()
	}
}

Go is advertising it can accept gzip, the server sends back gzip, and Go unzips the response and strips the Content-Length header before returning the response body. HEAD requests do not advertise gzip because there's no body to unzip and HTTP/1.1 headers can't be compressed.

It gets stripped here in net/http/transport.go:

		if rc.addedGzip && resp.Header.Get("Content-Encoding") == "gzip" {
			resp.Body = &gzipReader{body: body}
			resp.Header.Del("Content-Encoding")
			resp.Header.Del("Content-Length")
			resp.ContentLength = -1
			resp.Uncompressed = true
		}

curl returns the "proper" value because it doesn't advertise unzipping by default, if you pass Accept-Encoding: gzip you notice the server Content-Length changes.

@kevinburke

This comment has been minimized.

Copy link
Contributor

commented Feb 17, 2017

see also the comment about "Uncompressed" on a net/http.Response:

    // Uncompressed reports whether the response was sent compressed but
    // was decompressed by the http package. When true, reading from
    // Body yields the uncompressed content instead of the compressed
    // content actually set from the server, ContentLength is set to -1,
    // and the "Content-Length" and "Content-Encoding" fields are deleted
    // from the responseHeader. To get the original response from
    // the server, set Transport.DisableCompression to true.
    Uncompressed bool
@smasher164

This comment has been minimized.

Copy link
Contributor Author

commented Feb 17, 2017

Ah I see. I don't know how I missed the Uncompressed flag in http.Response. Thank You! I'll make sure to check these issues with golang-nuts next time before filing an "issue."

@smasher164 smasher164 closed this Feb 17, 2017

@golang golang locked and limited conversation to collaborators Feb 17, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.