https seems broken

[jdv]

Note that this:

$ perl6 -MHTTP::UserAgent -e 'say HTTP::UserAgent.new(timeout => 5).get("http://google.com").content.chars'
53781

works yet this:

$ perl6 -MHTTP::UserAgent -e 'HTTP::UserAgent.new(timeout=>5).get("https://google.com").content.chars.say'

just hung and had to be killed.

[sergot]

What OpenSSL's version do you have?

It works on my box:

$ perl6 -MHTTP::UserAgent -e 'HTTP::UserAgent.new(timeout => 5).get("https://google.com/").content.chars.say'
51789

$ openssl version
OpenSSL 1.0.1f 6 Jan 2014

btw. timeout is NYI for now

[jdv]

Its using libssl.so.0.9.8e from openssl-devel-0.9.8e-27.el5_10.4.i386.

From an strace it seems its stuck in an inf loop over read:

[pid 29911] read(14, "", 5) = 0

[ugexe]

it appears https forwarding is broken, likely since the URI commit. For example (github forwards to ssl as does the google example above):

# http -> https
nickl@localhost:~/perl6$ perl6 -e 'use HTTP::UserAgent; require IO::Socket::SSL; my $ua = HTTP::UserAgent.new; my $r = $ua.get("http://www.github.com/");'
err code: 336130315
error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number
err code: 336154692
Segmentation fault

# https directly
nickl@localhost:~/perl6$ perl6 -e 'use HTTP::UserAgent; require IO::Socket::SSL; my $ua = HTTP::UserAgent.new; my $r = $ua.get("https://www.github.com/");'
err code: 0
error:00000000:lib(0):func(0):reason(0)
err code: 0
error:00000000:lib(0):func(0):reason(0)

I say 'likely' because I had code that worked with https forwarding a few days ago (.get on http://www.github.com to be precise).

However if you .get a url that does not forward to https it works fine:

perl6 -e 'use HTTP::UserAgent; require IO::Socket::SSL; my $ua = HTTP::UserAgent.new; my $r = $ua.get("http://www.perl6.org/");'

[jdv]

This seems more like an issue with IO::Socket::SSL. I traced it down to the $!ssl.close call
hanging. Commenting it out seems to work. Not sure why its hanging though.

[ugexe]

nickl@localhost:~/perl6/zef$ perl6 -MIO::Socket::SSL -e 'my $c = IO::Socket::SSL.new(:host("github.com"), :port(443)); $c.send("GET /ugexe/zef/blob/master/META6.json HTTP/1.0\nHost: github.com\n\n"); say $c.recv'
Buf[uint8]:0x<48 54 54 50 2f 31 2e 31 20 32 30 30 20 4f 4b 0d 0a 53 65 72 76 65 72 3a 20 47 69 74 48 75 62 2e 63 6f 6d 0d 0a 44 61 74 65 3a 20 54 75 65 2c 20 31 33 20 4a 61 6e 20 32 30 31 35 20 31 38 3a 34 35 3a 34 31 20 47 4d 54 0d 0a 43 6f 6e 74 65 6e 74 2d 54 79 70 65 3a 20 74 65 78 74 2f 68 74 6d 6c 3b 20 63 68 61 72 73 65 74 3d 75 74 66 2d 38 0d 0a 43 6f 6e 6e 65 63 74 69 6f 6e 3a 20 63 6c 6f 73 65 0d 0a 53 74 61 74 75 73 3a 20 32 30 30 20 4f 4b 0d 0a 43 6f 6e 74 65 6e 74 2d 53 65 63 75 72 69 74 79 2d 50 6f 6c 69 63 79 3a 20 64 65 66 61 75 6c 74 2d 73 72 63 20 2a 3b 20 73 63 72 69 70 74 2d 73 72 63 20 61 73 73 65 74 73 2d 63 64 6e 2e 67 69 74 68 75 62 2e 63 6f 6d 20 63 6f 6c 6c 65 63 74 6f 72 2d 63 64 6e 2e 67 69 74 68 75 62 2e 63 6f 6d 3b 20 6f 62 6a 65 63 74 2d 73 72 63 20 61 73 73 65 74 73 2d 63 64 6e 2e 67 69 74 68 75 62 2e 63 6f 6d 3b 20 73 74 79 6c 65 2d 73 72 63 20 27 73 65 6c 66 27 20 27 75 6e 73 61 66 65 2d 69 6e 6c 69 6e 65 27 20 27 75 6e 73 61 66 65 2d 65 76 61 6c 27 20 61 73 73 65 74 73 2d 63 64 6e 2e 67 69 74 68 75 62 2e 63 6f 6d 3b 20 69 6d 67 2d 73 72 63 20 27 73 65 6c 66 27 20 64 61 74 61 3a 20 61 73 73 65 74 73 2d 63 64 6e 2e 67 69 74 68 75 62 2e 63 6f 6d 20 69 64 65 6e 74 69 63 6f 6e 73 2e 67 69 74 68 75 62 2e 63 6f 6d 20 77 77 77 2e 67 6f 6f 67 6c 65 2d 61 6e 61 6c 79 74 69 63 73 2e 63 6f 6d 20 63 6f 6c 6c 65 63 74 6f 72 2e 67 69 74 68 75 62 61 70 70 2e 63 6f 6d 20 2a 2e 67 69 74 68 75 62 75 73 65 72 63 6f 6e 74 65 6e 74 2e 63 6f 6d 20 2a 2e 67 72 61 76 61 74 61 72 2e 63 6f 6d 20 2a 2e 77 70 2e 63 6f 6d 3b 20 6d 65 64 69 61 2d 73 72 63 20 27 6e 6f 6e 65 27 3b 20 66 72 61 6d 65 2d 73 72 63 20 27 73 65 6c 66 27 20 72 65 6e 64 65 72 2e 67 69 74 68 75 62 75 73 65 72 63 6f 6e 74 65 6e 74 2e 63 6f 6d 20 67 69 73 74 2e 67 69 74 68 75 62 2e 63 6f 6d 20 77 77 77 2e 79 6f 75 74 75 62 65 2e 63 6f 6d 20 70 6c 61 79 65 72 2e 76 69 6d 65 6f 2e 63 6f 6d 20 63 68 65 63 6b 6f 75 74 2e 70 61 79 70 61 6c 2e 63 6f 6d 3b 20 66 6f 6e 74 2d 73 72 63 20 61 73 73 65 74 73 2d 63 64 6e 2e 67 69 74 68 75 62 2e 63 6f 6d 3b 20 63 6f 6e 6e 65 63 74 2d 73 72 63 20 27 73 65 6c 66 27 20 67 68 63 6f 6e 64 75 69 74 2e 63 6f 6d 3a 32 35 30 33 35 20 6c 69 76 65 2e 67 69 74 68 75 62 2e 63 6f 6d 20 75 70 6c 6f 61 64 73 2e 67 69 74 68 75 62 2e 63 6f 6d 20 77 77 77 2e 67 6f 6f 67 6c 65 2d 61 6e 61 6c 79 74 69 63 73 2e 63 6f 6d 20 73 33 2e 61 6d 61 7a 6f 6e 61 77 73 2e 63 6f 6d 0d 0a 43 61 63 68 65 2d 43 6f 6e 74 72 6f 6c 3a 20 6e 6f 2d 63 61 63 68 65 0d 0a 56 61 72 79 3a 20 58 2d 50 4a 41 58 0d 0a 58 2d 55 41 2d 43 6f 6d 70 61 74 69 62 6c 65 3a 20 49 45 3d 45 64 67 65 2c 63 68 72 6f 6d 65 3d 31 0d 0a 53 65 74 2d 43 6f 6f 6b 69 65 3a 20 6c 6f 67 67 65 64 5f 69 6e 3d 6e 6f 3b 20 64 6f 6d 61 69 6e 3d 2e 67 69 74 68 75 62 2e 63 6f 6d 3b 20 70 61 74 68 3d 2f 3b 20 65 78 70 69 72 65 73 3d 53 61 74 2c 20 31 33 2d 4a 61 6e 2d 32 30 33 35 20 31 38 3a 34 35 3a 34 31 20 47 4d 54 3b 20 73 65 63 75 72 65 3b 20 48 74 74 70 4f 6e 6c 79 0d 0a 53 65 74 2d 43 6f 6f 6b 69 65 3a 20 5f 67 68 5f 73 65 73 73 3d 65 79 4a 7a 5a 58 4e 7a 61 57 39 75 58 32 6c 6b 49 6a 6f 69 4e 6a 51 79 4d 6d 52 69 4e 6d 4d 79 59 57 4a 6c 4e 47 49 78 5a 6a 4a 6c 4e 57 49 35 4d 47 4d 30 4e 47 56 6a 5a 44 59 32 4d 6a 51 69 4c 43 4a 7a 63 48 6c 66 63 6d 56 77 62 79 49 36 49 6e 56 6e 5a 58 68 6c 4c 33 70 6c 5a 69 49 73 49 6e 4e 77 65 56 39 79 5a 58 42 76 58 32 46 30 49 6a 6f 78 4e 44 49 78 4d 54 63 30 4e 7a 51 78 4c 43 4a 66 59 33 4e 79 5a 6c 39 30 62 32 74 6c 62 69 49 36 49 6e 56 5a 53 55 39 75 54 46 49 31 64 33 6f 34 61 56 52 6a 63 56 70 70 62 55 70 59 63 44 6c 52 63 57 68 7a 4d 33 68 7a 61 32 31 49 63 6c 4d 33 64 7a 56 54 65 69 74 4f 57 6b 45 39 49 6e 30 25 33 44 2d 2d 34 63 33 38 33 62 63 31 33 62 37 33 35 36 30 38 36 37 33 64 32 65 66 38 31 63 64 62 31 66 63 32 37 61 35 36 65 31 32 35 3b 20 70 61 74 68 3d 2f 3b 20 73 65 63 75 72 65 3b 20 48 74 74 70 4f 6e 6c 79 0d 0a 58 2d 52 65 71 75 65 73 74 2d 49 64 3a 20 35 36 34 61 32 35 36 33 63 64 63 30 30 63 30 62 63 32 64 37 38 64 34 66 63 38 35 38 32 65 38 35 0d 0a 58 2d 52 75 6e 74 69 6d 65 3a 20 30 2e 30 33 30 38 31 36 0d 0a 58 2d 52 61 63 6b 2d 43 61 63 68 65 3a>

nickl@localhost:~/perl6/zef$ perl6 -MIO::Socket::SSL -e 'my $c = IO::Socket::SSL.new(:host("github.com"), :port(80)); $c.send("GET /ugexe/zef/blob/master/META6.json HTTP/1.0\nHost: github.com\n\n"); say $c.recv'
err code: 336130315
Segmentation fault

As you can see IO::Socket::SSL has no problem if it is given the correct port. HTTP::UserAgent gets the redirect via the headers 'location' field but fails to change the port.

http-useragent/lib/HTTP/UserAgent.pm6

Lines 196 to 197 in 23106d1

	last unless $response.status-line.substr(0, 1) eq '3' && $response.header.field('Location').defined;
	$url = ~$response.header.field('Location');

and here we see that while it matches the https and enters the correct block, it uses the default port if it is set:

http-useragent/lib/HTTP/UserAgent.pm6

Line 95 in 23106d1

$conn = ::('IO::Socket::SSL').new(:host(~$request.header.field('Host').values), :port($port // 443), :timeout($.timeout))

[jdv]

Those don't work for me for different reasons:

[jdv@jdv ~]$ perl6 -MIO::Socket::SSL -e 'my $c = IO::Socket::SSL.new(:host("github.com"), :port(443)); $c.send("GET /ugexe/zef/blob/master/META6.json HTTP/1.0\nHost: github.com\n\n"); say $c.recv'
err code: 336130329
error:1408F119:SSL routines:SSL3_GET_RECORD:decryption failed or bad record mac
err code: 0
error:00000000:lib(0):func(0):reason(0)
Buf[uint8]:0x<>
[jdv@jdv ~]$ perl6 -MIO::Socket::SSL -e 'my $c = IO::Socket::SSL.new(:host("github.com"), :port(80)); $c.send("GET /ugexe/zef/blob/master/META6.json HTTP/1.0\nHost: github.com\n\n"); say $c.recv'
err code: 336130315
error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number
err code: 336154692
error:14095044:SSL routines:SSL3_READ_N:internal error
Buf[uint8]:0x<>
[jdv@jdv ~]$

I have recent versions of stuff I think:

[jdv@jdv ~]$ rpm -qa openssl
openssl-0.9.8e-27.el5_10.4
[jdv@jdv ~]$ perl6 -v
This is perl6 version 2014.12-106-g97bf782 built on MoarVM version 2014.12-4-g7e95c05
[jdv@jdv ~]$

[ugexe]

nickl@localhost:~/perl6/zef$ openssl version
OpenSSL 1.0.1j 15 Oct 2014

nickl@localhost:~/perl6/zef$ perl6 -v
This is perl6 version 2014.12-118-gc5dcdfb built on MoarVM version 2014.12-4-g7e95c05

[sergot]

Is it ok now?

[jdv]

I can't test that because now Digest seems broken. The Digest tests hang and if
i install Digest with --notests then the tests for H::U hang:(

[jdv]

Now that Digest is fixed I can test and sadly report a different failure. One char
difference is all it takes:) See below.

[jdv@wieldy ~]$ perl6 -MHTTP::UserAgent -e 'say HTTP::UserAgent.new(timeout => 5).get("http://google.com").content.chars'
52687
[jdv@wieldy ~]$ perl6 -MHTTP::UserAgent -e 'say HTTP::UserAgent.new(timeout => 5).get("https://google.com").content.chars'
Too many positionals passed; expected 1 argument but got 2
in method request at lib/HTTP/UserAgent.pm6:118
in method get at lib/HTTP/UserAgent.pm6:87
in method get at lib/HTTP/UserAgent.pm6:90
in block at -e:1

[jdv@wieldy ~]$

[jonathanstowe]

@jdv this actually looks like something in IO::Socket::SSL, struggling to see where though.

[jonathanstowe]

Actually this might be a false hypothesis, I'm thinking that it might be to do with the redirects. If there is a relative redirect at certain points then the Host header doesn't get populated and boom.

[ugexe]

fwiw I added a .print method to IO::Socket::SSL a few days ago, which also happens to be called on the line of the error in question ($socket.print(...) where $socket may be IO::Socket::INET or IO::Socket::SSL). Before it would call my $socket = IO::Socket::SSL; $socket.print($x ~$y), but it had no such method (although from the error maybe its hidden in the native call stuff?)

[jonathanstowe]

Hi,
I think I have fixed it. The test url was redirecting to / and this resulted in a request without a host. I've refactored the way that redirects are handle such that if no Host can be dtermined from the Location header in the response the host port and scheme from the original request are used.

It's not very tidy but it seems to work now.

[jonathanstowe]

I think this is all good now, if someone else could check and close.

[jdv]

Confirmed fixed.