Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http: ReadResponse rejects duplicate Transfer-Encoding: chunked headers #44591

Open
andybalholm opened this issue Feb 24, 2021 · 5 comments
Open

Comments

@andybalholm
Copy link
Contributor

@andybalholm andybalholm commented Feb 24, 2021

Since Go 1.15 (commit d5734d4), http.ReadResponse (and therefore http.Transport.RoundTrip) returns an error if the response has more than one Transfer-Encoding header.

This makes it impossible to download content from certain sites. For example, attempting to download from https://sis.cat.com/ returns the error too many transfer encodings: ["chunked" "chunked"]. (https://play.golang.org/p/3rJb7ClBQ_U)

If I understand the discussion around the change correctly, there are security issues related to multiple Transfer-Encoding headers on a request. But this is on a response.

Could we be a little less strict for responses? Or allow multiple Transfer-Encoding headers if they are all "chunked"?

@thinkwelltwd
Copy link

@thinkwelltwd thinkwelltwd commented Feb 25, 2021

Here's another problematic URL:

https://onestop.md.gov

@fraenkel
Copy link
Contributor

@fraenkel fraenkel commented Mar 3, 2021

I did some further investigation.
The RFC is pretty clear on the matter but I was curious why curl and chrome didn't seem to care.

My testing included sending back multiple chunked headers on a single line as well as with multiple headers. I also sent back gzip, chunked variations which Go doesn't support.

Chrome didn't care what came back and always managed to print my response. They are just playing fast and loose to make the user experience appear to work in the face of all types of possible errors.

Curl didn't care how many chunked encodings were sent, but did fail when gzip, chunked didn't correctly encode the response body. Looking at their source code as well as some others, "chunked" is being scanned for and just marked as present. The additional encodings are then stacked. If one were to send chunked, gzip, curl doesn't care that its the wrong order.

Coalescing multiple chunked encodings won't hurt. The downside is that these broken configured servers are allowed to go on their merry way.

@thinkwelltwd
Copy link

@thinkwelltwd thinkwelltwd commented Mar 4, 2021

The downside is that these broken configured servers are allowed to go on their merry way.

That is vexing.

If curl and Chrome need to "make the user experience appear to work", and if the user can't between something that works vs "appears to work", should we as users of the Go library take that to mean that the world's waiting on the Go http library to enforce the spec?

Or could there be some other way to compile the Go library to "appear to work" as well as Chrome / curl? 🥺

As it currently is, I've downgraded to 1.14.14 and that doesn't seem like the best of alternatives either.

@neild
Copy link
Contributor

@neild neild commented Mar 21, 2021

@FiloSottile

Should we accept incorrectly duplicated "Transfer-Encoding: chunked" headers? This doesn't seem like it poses a request smuggling risk, although it does add a small amount of additional complexity to Transfer-Encoding parsing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
6 participants