Empty response body when Content-Length and Transfer-Encoding headers are omitted

[jknipp]

Beginning with v1.11.0, if the response from the server omits both the Content-Length and Transfer-Encoding response headers, the body of the response is not parsed and returned.

v1.11.0 Response Body

v1.10.1 Response Body

I believe the bug was introduced with the change in #450 to handle empty/invalid content-length.

5b51fb8#diff-03059c69a9df088e2476724af69f06bdL334

[intentionally-left-nil]

This is a breaking bug for us, we've had to pin our hackney at an old version to mitigate it.

[ConnorRigby]

I just ran into this issue also with Httpoison

[benoitc]

@jknipp good catch!

Is the server sending an HTTP 1.0 response ? 1.1 normally expects to have a content length. Can you check it?

I propose indeed to fix it by checking if the client receive an http 1.0 response and then read until a max. Thoughts?

[intentionally-left-nil]

@benoitc: Unfortunately, a lot of the internet doesn't follow specifications (HTTP 1.1 or otherwise). It would be beneficial for a client library to be as flexible as possible in this regard.

The spec actually doesn't require Content-length to be set. Instead, if the header is not sent, then the client must close the connection after it is completed. See the spec: https://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.4

5.By the server closing the connection. (Closing the connection cannot be used to indicate the end of a request body, since that would leave no possibility for the server to send back a response.)

[benoitc]

this is only for backward compatibility. I will only set a max to receive then. Fix release will land today.

…

Sent from my iPhone

On 27 Feb 2018, at 01:12, Anil Kulkarni ***@***.***> wrote: @benoitc: Unfortunately, a lot of the internet doesn't follow specifications (HTTP 1.1 or otherwise). It would be beneficial for a client library to be as flexible as possible in this regard. The spec actually doesn't require Content-length to be set. Instead, if the header is not sent, then the client must close the connection after it is completed. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[ericmj]

The server is following the spec, as @AnilRedshift said a missing content-length is valid and closing the connection will signal the end of the response body.

[benoitc]

well the server should do its best to send a content-length. I will fix it anyway.

[benoitc]

can you test #481 ? It shoudl fix your issue. Let me know :)

[PatNowak]

Hi,

I just tested. Value in response is correct, but the whole response is returned as an "error response". I use email decryptor service, so body returned is an email. My tests are in Elixir IEx.
Please take a look:

Response from hackney 1.10.0 via HTTPoison (default use case)

{:ok,
	%HTTPoison.Response{
	body: "[\"empty11@o3.pl\"]", 
	headers: headers,
	request_url: "URL",
	status_code: 200
}}

Response from hackney 1.11.0 (master) via HTTPoison - reason for this issue

{:ok,
	%HTTPoison.Response{
	body: "",
	headers: headers,
	request_url: "URL",
	status_code: 200
}}

Response from hackney from branch fix/gh-475 via HTTPoison

{:error, %HTTPoison.Error{id: nil, reason: {:closed, "[\"empty11@o3.pl\"]"}}}

Response from hackney from fix branch

{:ok, 200, _list, ref} = :hackney.request(:get, url, headers) # that's fine
 :hackney.body(ref)
{:error, {:closed, "[\"empty11@o3.pl\"]"}}

As you may see, the value is correct, but returns in error response.

[benoitc]

@PatNowak do you have a link I could test by chance? Can be done in private if you want to

[PatNowak]

It was url from private network, so it will not work for you. I can try to find an example that will be available publicly :)

Edit: Sorry, I didn't find an example in public network :/

[benoitc]

@PatNowak I pushed ce32d21 in the PR that should fix your issue. Let me know :)

[benoitc]

@PatNowak ?

[benoitc]

bump.

[filmor]

@benoitc, in the meantime, would you please retract the package from Hex?

[benoitc]

@filmor Only mix has this feature. Anyway did you test the PR, it would help to have some feedback...

[filmor]

I think you can ping the hex admins (support@hex.pm), I'll test the PR on Monday.

[benoitc]

@filmor ping?