Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http: RoundTrip unexpectedly changes Request #39533

Closed
neild opened this issue Jun 11, 2020 · 7 comments
Closed

net/http: RoundTrip unexpectedly changes Request #39533

neild opened this issue Jun 11, 2020 · 7 comments
Assignees
Milestone

Comments

@neild
Copy link
Contributor

@neild neild commented Jun 11, 2020

https://go-review.googlesource.com/c/go/+/234894 causes (*http.Transport).RoundTrip to return a different *http.Request in the *http.Response than was passed to it, under some circumstances.

func main() {
        req, err := http.NewRequest("GET", "http://google.com/", bytes.NewBuffer([]byte{0, 1, 2}))
        if err != nil {
                log.Fatal(err)
        }
        resp, err := http.DefaultTransport.RoundTrip(req)
        if err != nil {
                log.Fatal(err)
        }
        fmt.Println(req == resp.Request) // false, where it used to be true.
}

The documentation for (http.Response).Request says, "Request is the request that was sent to obtain this Response." While this doesn't precisely specify that it's the exact *Request passed to RoundTrip, that's probably close enough to consider this an incorrect API change.

Discovered because this breaks code which keeps a map of *http.Requests.

@neild
Copy link
Contributor Author

@neild neild commented Jun 11, 2020

Even prior to this change, RoundTrip would change the Request, but only when calling GetBody to rewind the body:
https://go.googlesource.com/go/+/0e617d3d5c7e89b1ad1b0285fc77314b8d056211/src/net/http/transport.go#588

Tracking the history of this behavior, I believe it originates with https://golang.org/cl/33971, which causes the HTTP2 transport to replace the request when retrying.

My first thought was that it seems odd to replace the entire *Request rather than just updating its Body field, but the http.RoundTripper documentation states:

RoundTrip should not modify the request, except for consuming and closing the Request's Body.

Replacing the *Request in RoundTrip only under unusual circumstances (when calling GetBody to reset the body) is error-prone; users may depend on the *Request passing through unchanged, and only encounter infrequent errors. I think the *Request should be replaced consistently, or never.

My inclination is to say that "never" is the right answer, since it causes the least disruption to existing users.

If that's the case, we could:

  • Change the RoundTripper contract to permit changing the request Body when GetBody is called. I think this is reasonable behavior, but I'm not certain if this would be consistent with our compatibility guarantees.
  • Temporarily replace the request Body, but put the original one back before returning.
  • Create a new Request when retrying, but return the original in the Response.
@rsc
Copy link
Contributor

@rsc rsc commented Jun 11, 2020

Requests are read-only data, immutable. I believe it is allowed today to call RoundTrip from multiple goroutines using the same request (provided Body = NoBody or nil). Mutating the Request would break that, introducing races.

The main time that Requests change and the reason Response records the final Request is when http.Client.Do follows redirects. That's the time when you most often need to look at the Response - to find out what the final fetched URL even was (if you care). Since they can change in Do, I'm not convinced it's important they can't change in RoundTrip.

Maybe we can say that the Response returned by RoundTrip always returns the original Request? I'm not sure. It's either that or decide the tests are broken. You can't mutate in place.

/cc @bradfitz

@gopherbot
Copy link

@gopherbot gopherbot commented Jun 11, 2020

Change https://golang.org/cl/237560 mentions this issue: net/http: make Transport.RoundTrip preserve Requests

@neild
Copy link
Contributor Author

@neild neild commented Jun 11, 2020

We'd only need to mutate the Request if the Body is neither NoBody nor nil, which are the cases where you aren't allowed to call RoundTrip from multiple goroutines. If immutable requests are an invariant, though, then it is what it is.

@dmitshur
Copy link
Member

@dmitshur dmitshur commented Aug 22, 2020

@gopherbot Please backport to Go 1.14.

This is a fixup to a previous fix (CL 242117) for an approved backport issue #39279. We can't submit that CL without this one. This needs to be backported to Go 1.14 only, since the fix is already included in Go 1.1​5.

@gopherbot
Copy link

@gopherbot gopherbot commented Aug 22, 2020

Backport issue(s) opened: #40973 (for 1.14).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.

@gopherbot
Copy link

@gopherbot gopherbot commented Aug 22, 2020

Change https://golang.org/cl/249880 mentions this issue: [release-branch.go1.14] net/http: make Transport.RoundTrip preserve Requests

gopherbot pushed a commit that referenced this issue Aug 27, 2020
…equests

Ensure that the exact Request passed to Transport.RoundTrip
is returned in the Response. Do not replace the Request with
a copy when resetting the request body.

Updates #39533.
Fixes #40973.

Change-Id: Ie6fb080c24b0f6625b0761b7aa542af3d2411817
Reviewed-on: https://go-review.googlesource.com/c/go/+/237560
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>
Reviewed-on: https://go-review.googlesource.com/c/go/+/249880
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Damien Neil <dneil@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants
You can’t perform that action at this time.