Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

net/http: client/transport should be more defensive against servers which write a body in response to HEAD requests #68653

Open
uhthomas opened this issue Jul 30, 2024 · 4 comments
Open

net/http: client/transport should be more defensive against servers which write a body in response to HEAD requests #68653

uhthomas opened this issue Jul 30, 2024 · 4 comments
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Backlog

Comments

@uhthomas
Copy link

uhthomas commented Jul 30, 2024 •
edited
Loading

Go version

go version go1.22.5 linux/amd64

Output of go env in your module/workspace:

N/A

What did you do?

The Go HTTP client/transport may reuse connections which have been used for a HEAD request. If the server wrote a body to the response, the Go HTTP client/transport may use that body as if it were the start of a new request.

Ref: #68609

What did you see happen?

attempt 0
2024/07/26 16:17:17 Unsolicited response received on idle HTTP channel starting with "<!DOCTYPE html>\n<html>\n  <head>\n    <title>Thomas</tit"; err=<nil>
attempt 1
2024/07/26 16:17:17 Unsolicited response received on idle HTTP channel starting with "<!DOCTYPE html>\n<html>\n  <head>\n    <title>Thomas</tit"; err=<nil>
attempt 2
2024/07/26 16:17:17 Unsolicited response received on idle HTTP channel starting with "<!DOCTYPE html>\n<html>\n  <head>\n    <title>Thomas</tit"; err=<nil>
attempt 3
2024/07/26 16:17:17 Unsolicited response received on idle HTTP channel starting with "<!DOCTYPE html>\n<html>\n  <head>\n    <title>Thomas</tit"; err=<nil>
attempt 4
2024/07/26 16:17:17 Unsolicited response received on idle HTTP channel starting with "<!DOCTYPE html>\n<html>\n  <head>\n    <title>Thomas</tit"; err=<nil>
attempt 5
2024/07/26 16:17:17 Unsolicited response received on idle HTTP channel starting with "<!DOCTYPE html>\n<html>\n  <head>\n    <title>Thomas</tit"; err=<nil>
attempt 6
attempt 7
2024/07/26 16:17:18 do: Head "http://localhost:8080": net/http: HTTP/1.x transport connection broken: malformed HTTP status code "html>"
exit status 1

What did you expect to see?

The Go HTTP client/transport should be more defensive and not get tripped up on servers misbehaving like this. It's not uncommon for this to happen, as noted in #68609 (comment). It's quite problematic as there are lots of reverse proxies which rely on the net/http transport (cloudflared, caddy, etc).

image

image
@uhthomas uhthomas mentioned this issue Jul 30, 2024
Closed
@uhthomas uhthomas changed the title net/http: Client/Transport should be more defensive against servers which write a body in response to HEAD requests net/http: client/transport should be more defensive against servers which write a body in response to HEAD requests Jul 30, 2024
@gabyhelp
Copy link

Related Issues and Documentation

(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)

@mknyszek
Copy link
Contributor

CC @neild

@mknyszek mknyszek added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jul 31, 2024
@mknyszek mknyszek added this to the Backlog milestone Jul 31, 2024
@neild
Copy link
Contributor

neild commented Jul 31, 2024

What do you think the client should do that it is not doing now?

If the server sends a body in response to a HEAD request, then:

  • If the client observes the body before reusing the connection for another request, it closes the connection and logs an error: "Unsolicited response received...".
  • If the client receives the body after sending another request on the connection, it cannot distinguish between the body and a malformed response to the new request. However, since the body (usually) doesn't match the syntax of a response, it closes the connection and logs an error: "malformed HTTP status code..."

There's no way for the client to ensure that a server hasn't improperly sent a body in response to a HEAD request: You can't prove a negative. We could be a bit more aggressive in checking for data in a conn's read buffer, but if the server has sent response headers and body in separate packets, that isn't going to be very reliable--we'll still hit the same error paths we do today when we process headers and haven't yet received the body.

Perhaps our error logging could be clearer.

But it isn't clear to me what "more defensive" would be, specifically, or in what way the client is currently "getting tripped up on" misbehaving servers.

@uhthomas
Copy link
Author

Thanks for following up @neild. Following from what @terinjokes said, does it make sense to close connections after a HEAD request?

#68609 (comment)

Maybe I'm reading it wrong, but this RFC suggests that it may be advisable.

Although request message framing is independent of the method used, content received in a HEAD request has no generally defined semantics, cannot alter the meaning or target of the request, and might lead some implementations to reject the request and close the connection because of its potential as a request smuggling attack (Section 11.2 of [HTTP/1.1]).

https://httpwg.org/specs/rfc9110.html#HEAD

What are your thoughts?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Milestone
Backlog
Development

No branches or pull requests
4 participants
@neild @mknyszek @uhthomas @gabyhelp