Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http: client will only send a single user-agent header value #41738

Closed
mdwhatcott opened this issue Oct 1, 2020 · 4 comments
Closed

net/http: client will only send a single user-agent header value #41738

mdwhatcott opened this issue Oct 1, 2020 · 4 comments

Comments

@mdwhatcott
Copy link

@mdwhatcott mdwhatcott commented Oct 1, 2020

What version of Go are you using (go version)?

$ go version
go version go1.15.2 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/mike/Library/Caches/go-build"
GOENV="/Users/mike/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/mike/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/mike"
GOPRIVATE=""
GOPROXY="direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/mv/rt57xj7n1xl1wvqn5h0y2x4m0000gp/T/go-build718939165=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

request, _ := http.NewRequest("GET", "http://localhost:8080/", nil)
request.Header.Add("Content-Type", "A")
request.Header.Add("Content-Type", "B")
request.Header.Add("User-Agent", "C")
request.Header.Add("User-Agent", "D")
_, _ = http.DefaultClient.Do(request)

What did you expect to see?

I expected the server to receive two Content-Type values ("A" & "B") and two User-Agent values ("C" & "D").

What did you see instead?

The server did receive two Content-Type values as expected, but only the first User-Agent value ("C") was received.

Is this a bug or oversight, or is there a good reason for the Go HTTP client to not send multiple user-agent values?


The http.Header type models HTTP headers, which can appear multiple times in the same request/response, by using a map[string][]string (note the []string value). The following snippet demonstrates that some headers, such as User-Agent are only sent once by the Go HTTP client in an outgoing HTTP request:

package main

import (
	"log"
	"net/http"
)

func main() {
	log.SetFlags(0)

	go func() { _ = http.ListenAndServe("localhost:8080", new(echo)) }()

	request, _ := http.NewRequest("GET", "http://localhost:8080/", nil)

	request.Header.Add("Content-Type", "A")
	request.Header.Add("Content-Type", "B")
	request.Header.Add("User-Agent", "C")
	request.Header.Add("User-Agent", "D") // Will not be sent!

	log.Println("Client:", request.Header)

	_, _ = http.DefaultClient.Do(request)
}

type echo struct{}

func (echo) ServeHTTP(_ http.ResponseWriter, request *http.Request) {
	request.Header.Del("Accept-Encoding")
	log.Println("Server:", request.Header)
}

The output of the above program is as follows:

Client: map[Content-Type:[A B] User-Agent:[C D]]
Server: map[Content-Type:[A B] User-Agent:[C]]

However, using other tools (such as cURL) it is possible to send multiple User-Agent headers:

$ curl -v "https://www.google.com" -H 'User-Agent: 1' -H 'User-Agent: 2'
...
> GET / HTTP/2
> Host: www.google.com
> Accept: */*
> User-Agent: 1
> User-Agent: 2
...

Why does Go disallow multiple User-Agent values?

For reference, I believe this is the code that limits the User-Agent to a single value:

https://github.com/golang/go/blob/master/src/net/http/h2_bundle.go#L8025-L8028

} else if strings.EqualFold(k, "user-agent") {
	// Match Go's http1 behavior: at most one
	// User-Agent. If set to nil or empty string,
	// then omit it. Otherwise if not mentioned,
	// include the default (below).
	didUA = true
	if len(vv) < 1 {
		continue
	}
	vv = vv[:1]
	if vv[0] == "" {
		continue
	}
}

Is this a bug or oversight, or is there a good reason for the Go HTTP client to not send multiple user-agent values?

@andybons
Copy link
Member

@andybons andybons commented Oct 1, 2020

@andybons andybons added this to the Unplanned milestone Oct 1, 2020
@bradfitz
Copy link
Contributor

@bradfitz bradfitz commented Oct 1, 2020

Looks pretty intentional at least (per that comment).

Also seems like reasonable behavior. Certain well-known headers only appear once.

Arguably Content-Type could be capped to just 1 too, for security reasons.

@fraenkel
Copy link
Contributor

@fraenkel fraenkel commented Oct 2, 2020

See https://tools.ietf.org/html/rfc7230#section-3.2.2
Sending duplicate headers is only valid if the values are comma separated.

@mdwhatcott
Copy link
Author

@mdwhatcott mdwhatcott commented Oct 2, 2020

@fraenkel - Thanks for pointing that out! I had been looking for some pronouncement like that but wasn't able to see it.

@mdwhatcott mdwhatcott closed this Oct 2, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants