net/http: can't read 301 response without a Location header

[stefansundin]

What version of Go are you using (go version)?

$ go version
go version go1.17.2 darwin/arm64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GOARCH="arm64"
GOOS="darwin"

What did you do?

I am trying to use net/http to send a request to https://example.s3.us-west-1.amazonaws.com/ and read the response in order to get the actual bucket location (the correct AWS region). This response is missing a Location header, so net/http just returns an error.

package main

import (
	"errors"
	"fmt"
	"net/http"
)

func getBucketRegion(bucket string) (string, error) {
	// Construct client that makes one request and does not follow redirects
	client := &http.Client{
		CheckRedirect: func(req *http.Request, via []*http.Request) error {
			return http.ErrUseLastResponse
		},
	}
	resp, err := client.Get("https://" + bucket + ".s3.us-west-1.amazonaws.com/")
	if err != nil {
		return "", err // <-- function will return here
	}

	if resp.StatusCode == 200 {
		return "us-west-1", nil
	} else if resp.StatusCode == 404 {
		return "", errors.New("Bucket does not exist.")
	}

	return resp.Header.Get("x-amz-bucket-region"), nil
}

func main() {
	region, err := getBucketRegion("test")
	fmt.Printf("region: %s\n", region)
	fmt.Printf("err: %v\n", err)
}

This does not run on play.golang.org because there is no network access. https://play.golang.org/p/BiEmt_5iEvJ

Output will be:

region:
err: Get "https://example.s3.us-west-1.amazonaws.com/": 301 response missing Location header

It is impossible to get the response to read the x-amz-bucket-region header and get the actual region. One could argue that S3 does not conform to HTTP specs by returning a 301 response code without a Location header. But Go's refusal to give me back the response means that I'm not able to request and handle this particular case.

It would be nice to just return the request in this case, since there is no redirect to follow.

https://cs.opensource.google/go/go/+/master:src/net/http/client.go;l=643-646;drc=aa4e0f528e1e018e2847decb549cfc5ac07ecf20

			if loc == "" {
				resp.closeBody()
				return nil, uerr(fmt.Errorf("%d response missing Location header", resp.StatusCode))
			}

A little bit below that, there's a place where both a response and an error is returned (the comment says that it is to maintain Go 1 compatibility). Perhaps a similar handling could be added here?

Something like:

			if loc == "" {
				return resp, uerr(fmt.Errorf("%d response missing Location header", resp.StatusCode))
			}

This is just an idea, I'm sure there may be a better way!

What did you expect to see?

$ curl -v https://example.s3.us-west-1.amazonaws.com/
[...]

< HTTP/1.1 301 Moved Permanently
< x-amz-bucket-region: us-east-1
[...]

I want the ability to get x-amz-bucket-region: us-east-1 header from the response. But since the Location header is missing it is not possible to use net/http to get it.

What did you see instead?

Get "https://example.s3.us-west-1.amazonaws.com/": 301 response missing Location header

[seankhliao]

You could use http.Transport.Roundtrip directly or add one that modifies the response to suit your needs

[seankhliao]

cc @neild

[neild]

RFC 7231 says that "the server SHOULD generate a Location header field in the response" for a 301 or 302. I'm not sure why we're treating it as an error for the Location header to be missing, since it is not required.

My first thought is that we should just return the response with no error for any 3xx status with no Location header.