ERL-316: httpc client redirection either malfunctioning or unclear

[OTP-Maintainer]

Original reporter: quickcheck
Affected versions: OTP-19.1.1, OTP-19.1
Fixed in version: OTP-19.3
Component: inets
Migrated from: https://bugs.erlang.org/browse/ERL-316

---

h2. Firefox and Safari behave different from HTTPC

When a webserver responses with 302, the client gets a new location to redirect the request to.
The httpc server has autoredirect by default set to true and following the redirects is expected.

While using an industrial web service, I used httpc to follow a link that was redirected. This did not work. The wireshark indicated that after the second redirect, the port was kind of lost and a request went out to localhost instead of localhost:8090.

I tried to find out what the reason is, but didn't get the whole way.
Attached is a little webserver based on httpd. Whenever we request page http://localhost:8090/0, we get "Welcome". If the Path is a digit larger than 0, say 2, then the page is redirected to N-1.
Thus, localhost:8090/2 is redirected with status 302 to localhost:8090/1 and then to localhost:8090/0, resulting in the "Welcome" page.

Both Safari and Firefox (both on Mac), do understand this redirection and get to the Welcome page.

When I use httpc, I get a timeout. This timeout differs from the error message I got originally, viz, getting redirected to localhost/... instead of localhost:8090/...., thus I may be doing something wrong there.

But even a timeout is weird. Even when I do set {autoredirect, false}, then I get a timeout.
Tracing the wire shows differences in TCP, but the response comes back from the webservice.

The http request is stuck in the receive loop starting at line 621 of httpc.erl

Appreciated if a Web Server expert can give me some hints to dig deeper into this.

[OTP-Maintainer]

ingela said:

Will the following commit 9aaadfcc448b40a6cf3bc567b5949c4ee6ac04a2 perhaps solve your problem?

[OTP-Maintainer]

quickcheck said:

I have compiled and installed Erlang OTP-19.1.6, which is newer than the patch you point to.

The attached Erlang program is a good test case. Please check whether it works for you.

In Erlang shell:
Erlang/OTP 19 [erts-8.1.1] [source] [64-bit] [smp:4:4] [async-threads:10] [hipe] [kernel-poll:false]

Eshell V8.1.1  (abort with ^G)
1> webserver:start().
{ok,<0.72.0>}

Firefox or other browser: http://localhost:8090/0
See a "Welcome" message
Firefox or other browser: http://localhost:8090/4
See that the address changes to localhost:8090/0 and "Welcome message" is there.

Now trying to emulate Firefox with httpc:
2> webserver:test(0).
{ok,{{"HTTP/1.1",200,"OK"},
     [{"date","Mon, 12 Dec 2016 11:36:32 GMT"},
      {"server","inets/6.3.3"},
      {"content-length","7"},
      {"content-type","text/html"}],
     "Welcome"}}
3> webserver:test(1).
{error,timeout}

Or if you want to try directly on httpc:
4> httpc:request("http://localhost:8090/0").
{ok,{{"HTTP/1.1",200,"OK"},
     [{"date","Mon, 12 Dec 2016 11:39:13 GMT"},
      {"server","inets/6.3.3"},
      {"content-length","7"},
      {"content-type","text/html"}],
     "Welcome"}}
5> httpc:request("http://localhost:8090/1").
...hanging...

[OTP-Maintainer]

quickcheck said:

and I pushed the wrong button on... it's still assigned to you :(

[OTP-Maintainer]

ingela said:

I am not sure what you mean by earlier. The commit I mentioned is included for OTP-19.2. Anyhow your test program works for me if you change 
it to also send a content-length header.

[OTP-Maintainer]

quickcheck said:

Thanks Ingela

That solved the first problem, but there is still at least one bug.
I pulled git and re-build from master branch with freshed commit: e1cdd3732f5ff447195625620b78883c2f22fb65

As you pointed out, I needed to update my webserver. Even though Firfow, Chrome and Safari do accept it as it is.
{code}webserver.erl
     [Nr] ->
      {proceed, [{response, {response,
                             [{location, lists:concat(["http://",Host,":",Port,"/",[Nr-1]])},
                              {content_length, "0"},
                              {code, 302}
                             ], 
                             []}}]}{code}

Now it works for 1 redirect, but not for 2, at least not for me.

{code}42> webserver:test(1).
{ok,{{"HTTP/1.1",200,"OK"},
     [{"date","Mon, 12 Dec 2016 18:56:14 GMT"},
      {"server","inets/6.3.3"},
      {"content-length","7"},
      {"content-type","text/html"}],
     "Welcome"}}
43> webserver:test(4).
{error,{failed_connect,[{to_address,{"localhost",80}},
                        {inet,[inet],econnrefused}]}}
44> webserver:test(2).
{error,{failed_connect,[{to_address,{"localhost",80}},
                        {inet,[inet],econnrefused}]}}

{code}

Indeed when you look at the HTTP messages at Wireshark or trace them, then you see that the scond redirect points the user to port 80 instead of port 8090.

[OTP-Maintainer]

quickcheck said:

And I found the error:

in http_response.erl line 366 the new header 'host' is created from only the host, but in the headers it should also include the port. A dirty hack to show that this is the case:

{code}NewHeaders = 
	(Request#request.headers)#http_request_h{host = Host++":"++integer_to_list(Port)},{code}

RFC2616 Sect 14.23: The Host request-header field specifies the Internet host AND port number.

[OTP-Maintainer]

ingela said:

We will fix it for 19.3

[OTP-Maintainer]

voltone said:

The new behaviour in 19.3 of always including a port number, even for well known ports 80 and 443, can cause new issues. For instance, services that redirect to S3 using signed requests assume the client will not include :443 in the Host header:
https://github.com/nerves-project/nerves/issues/96

Technically I guess S3 is at fault here for not normalising the Host header when signing and verifying the request, but I think it would be best for `httpc` to mimic the behaviour of other clients: omit the port number if it is the well known port for the given scheme.

I'd be happy to create a PR for that. Should I open a new issue here too?

[OTP-Maintainer]

ingela said:

You do not need to create an issue here if you make a PR.

[OTP-Maintainer]

voltone said:

OK, thanks. I made this: https://github.com/erlang/otp/pull/1381