Ensure response to HEAD request won't have message body

[darkrain42]

Ensure that Gunicorn won't try to use chunked transfer-encoding for responses
to a HEAD request, so that Response.close will not write a terminating
chunk. Responses to a HEAD request MUST NOT have a message-body.

The application is still responsible for ensuring no message body is actually
generated in response to a HEAD request.

Refs #990

[benoitc]

thanks!

[jcea]

I have spent two months investigating a sporadic malfunction of an application of mine. I finally found this PR and it solved my problem.

My problem was HEAD leaving "garbage" behind in the socket "sometimes", that Apache (acting as a reverse proxy) was reusing as the response to the next request.

Could you consider pushing a new gunicorn release?

[benoitc]

there is one planned next week. However if you're returning some data on head, better fix your app since it will likely try to send a body when gunicorn will ignore it :)

[jcea]

I do not return anything in HEAD. Just "return []".

Looking forward the new release :)

[benoitc]

well if you only return that the patch shouldnt change anything. anyway will update you when the release is done.

[jcea]

The patch solves my problem because "HEAD" was sending a chunked fragment of size zero when the standard says that HEAD doesn't return ANY body at all. The result was that HEAD was ok, but next request reusing the connection will read the "pending" chunked reply as the beginning of a HTTP/0.9 reply, with bad results.

So, yes, this patch is actually critical.

Steps to reproduce:

1. Reuse the same connection for multiple requests.
2. Send a HEAD request with an undefined reply length. That is, no "content-length" header.
3. Observe that HEAD returns a chunked reply with a chunk fragment of size 0. THAT IS WRONG, and that is what this patch solves.
4. An Apache HTTP server frontend will read the HEAD reply UP TO the chunked reply, because it is not supposed to be there.
5. Send a new request thru the same connection.
6. Apache will read the reply... including the chunked header at the beginning.
7. The second request will get a mangled reply.

[darkrain42]

I think you're both agreeing, but talking about what different parts of the stack are sending. @jcea is talking about what Gunicorn send back to the client (Apache, in this case), and @benoitc is talking about what the app "sends" to Gunicorn.

@jcea, your issue was exactly the same as what I was troubleshooting until I stumbled across a reproduction and isolated it. I ended up swapping out Apache for nginx, which also resolved the issue as well -- nginx behaves better in the face of the response to the HEAD request having a content-body.

[jcea]

@darkrain42, then nginx is defective :-p. Anyway, thanks for catching and solving this!

[jcea]

One of the steps I did to triage this was to configure Apache to avoid connection reusing to the gunicorn backend. That also solved the perceived issue but, of course, not the real issue.