net/http: can't read 301 response without a Location header #49281
Description
What version of Go are you using (go version)?
$ go version go version go1.17.2 darwin/arm64
Does this issue reproduce with the latest release?
Yes.
What operating system and processor architecture are you using (go env)?
go env Output
$ go env GOARCH="arm64" GOOS="darwin"
What did you do?
I am trying to use net/http to send a request to https://example.s3.us-west-1.amazonaws.com/ and read the response in order to get the actual bucket location (the correct AWS region). This response is missing a Location header, so net/http just returns an error.
package main
import (
"errors"
"fmt"
"net/http"
)
func getBucketRegion(bucket string) (string, error) {
// Construct client that makes one request and does not follow redirects
client := &http.Client{
CheckRedirect: func(req *http.Request, via []*http.Request) error {
return http.ErrUseLastResponse
},
}
resp, err := client.Get("https://" + bucket + ".s3.us-west-1.amazonaws.com/")
if err != nil {
return "", err // <-- function will return here
}
if resp.StatusCode == 200 {
return "us-west-1", nil
} else if resp.StatusCode == 404 {
return "", errors.New("Bucket does not exist.")
}
return resp.Header.Get("x-amz-bucket-region"), nil
}
func main() {
region, err := getBucketRegion("test")
fmt.Printf("region: %s\n", region)
fmt.Printf("err: %v\n", err)
}This does not run on play.golang.org because there is no network access. https://play.golang.org/p/BiEmt_5iEvJ
Output will be:
region:
err: Get "https://example.s3.us-west-1.amazonaws.com/": 301 response missing Location header
It is impossible to get the response to read the x-amz-bucket-region header and get the actual region. One could argue that S3 does not conform to HTTP specs by returning a 301 response code without a Location header. But Go's refusal to give me back the response means that I'm not able to request and handle this particular case.
It would be nice to just return the request in this case, since there is no redirect to follow.
if loc == "" {
resp.closeBody()
return nil, uerr(fmt.Errorf("%d response missing Location header", resp.StatusCode))
}A little bit below that, there's a place where both a response and an error is returned (the comment says that it is to maintain Go 1 compatibility). Perhaps a similar handling could be added here?
Something like:
if loc == "" {
return resp, uerr(fmt.Errorf("%d response missing Location header", resp.StatusCode))
}This is just an idea, I'm sure there may be a better way!
What did you expect to see?
$ curl -v https://example.s3.us-west-1.amazonaws.com/
[...]
< HTTP/1.1 301 Moved Permanently
< x-amz-bucket-region: us-east-1
[...]
I want the ability to get x-amz-bucket-region: us-east-1 header from the response. But since the Location header is missing it is not possible to use net/http to get it.
What did you see instead?
Get "https://example.s3.us-west-1.amazonaws.com/": 301 response missing Location header