Skip to content
This repository

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Two small fixes. #18

Merged
merged 4 commits into from over 1 year ago

2 participants

ronaldxs Cosimo Streppone
ronaldxs

Fixed a test that I made too complicating because some confusion on encodings.

Updated readme. AFAIK binary works now.

ronaldxs added some commits
ronaldxs ronaldxs Update README
Removed note about corrupting binary files.  Should not do that any more.
2b40128
ronaldxs ronaldxs Update t/get-w3-latin1-utf8.t
Simpler, less confusing, test code.  Fancy buf decoding did nothing useful.
7d798b7
ronaldxs ronaldxs Fix and test for case with no content length or chunking 149f3ce
ronaldxs ronaldxs better test description 9bfeb2d
ronaldxs

Added fix for significant bug that prevents whole page from being pulled when server does not provide Content-length or do chunking. Developed and tested with 2012-09 version of Rakudo Star because of some known concerns with LWP::Simple and recent Rakudo commits.

Cosimo Streppone cosimo merged commit 40da36d into from
Cosimo Streppone cosimo closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Showing 4 unique commits by 1 author.

Aug 30, 2012
ronaldxs ronaldxs Update README
Removed note about corrupting binary files.  Should not do that any more.
2b40128
ronaldxs ronaldxs Update t/get-w3-latin1-utf8.t
Simpler, less confusing, test code.  Fancy buf decoding did nothing useful.
7d798b7
Oct 22, 2012
ronaldxs ronaldxs Fix and test for case with no content length or chunking 149f3ce
ronaldxs ronaldxs better test description 9bfeb2d
This page is out of date. Refresh to see the latest.
2  README
@@ -22,5 +22,3 @@ Current status
22 22 As of 2011-04-22, runs with all recent rakudo builds.
23 23 It correctly follows redirects, but no infinite redirects
24 24 detection yet.
25   -
26   -A known problem is that it corrupts binary files downloads.
24 lib/LWP/Simple.pm
@@ -17,6 +17,7 @@ our $.class_default_encoding = 'utf-8';
17 17 # these were intended to be constant but that hit pre-compilation issue
18 18 my Buf $crlf = Buf.new(13, 10);
19 19 my Buf $http_header_end_marker = Buf.new(13, 10, 13, 10);
  20 +my Int constant $default_stream_read_len = 2 * 1024;
20 21
21 22 method base64encode ($user, $pass) {
22 23 my MIME::Base64 $mime .= new();
@@ -194,19 +195,11 @@ method make_request (
194 195
195 196 $sock.send($req_str);
196 197
197   - my Buf $resp = $sock.read(2 * 1024);
  198 + my Buf $resp = $sock.read($default_stream_read_len);
198 199
199 200 my ($status, $resp_headers, $resp_content) = self.parse_response($resp);
200 201
201 202
202   - if ( $resp_headers<Content-Length> &&
203   - $resp_content.bytes < $resp_headers<Content-Length>
204   - ) {
205   - $resp_content ~= $sock.read(
206   - $resp_headers<Content-Length> - $resp_content.bytes
207   - );
208   - }
209   -
210 203 if (($resp_headers<Transfer-Encoding> || '') eq 'chunked') {
211 204 my Bool $is_last_chunk;
212 205 my Buf $resp_content_chunk;
@@ -222,6 +215,19 @@ method make_request (
222 215 $resp_content ~= $resp_content_chunk;
223 216 }
224 217 }
  218 + elsif ( $resp_headers<Content-Length> &&
  219 + $resp_content.bytes < $resp_headers<Content-Length>
  220 + ) {
  221 + $resp_content ~= $sock.read(
  222 + $resp_headers<Content-Length> - $resp_content.bytes
  223 + );
  224 + }
  225 + else { # a bit hacky for now but should be ok
  226 + while ($resp.bytes > 0) {
  227 + $resp = $sock.read($default_stream_read_len);
  228 + $resp_content ~= $resp;
  229 + }
  230 + }
225 231
226 232 $sock.close();
227 233
16 t/get-unsized.t
... ... @@ -0,0 +1,16 @@
  1 +use v6;
  2 +use Test;
  3 +
  4 +use LWP::Simple;
  5 +
  6 +# this page is, for now, delivered by a server that does not provide
  7 +# a content length or do chunking
  8 +my $html = LWP::Simple.get('http://www.rosettacode.org/wiki/Rosetta_Code');
  9 +
  10 +ok(
  11 + $html.match('About Rosetta Code') &&
  12 + $html.match('</html>') && $html.chars > 12_000,
  13 + 'make sure we pulled whole document without, we believe, sizing from server'
  14 +);
  15 +
  16 +done;
3  t/get-w3-latin1-utf8.t
@@ -5,14 +5,13 @@ use LWP::Simple;
5 5
6 6 my $html = LWP::Simple.get('http://www.w3.org/2006/11/mwbp-tests/test-encoding-8.html');
7 7
8   -my $find_char = Buf.new(0xE9).decode('iso-8859-1');
  8 +my $find_char = chr(233); # small e with acute
9 9 ok(
10 10 $html.match('</html>') && $html.match($find_char),
11 11 'Got latin-1 page'
12 12 );
13 13
14 14 $html = LWP::Simple.get('http://www.w3.org/2006/11/mwbp-tests/test-encoding-3.html');
15   -$find_char = Buf.new(0xC3, 0xA9).decode('utf-8');
16 15 ok(
17 16 $html.match('</html>') && $html.match($find_char),
18 17 'Got utf-8 page'

Tip: You can add notes to lines in a file. Hover to the left of a line to make a note

Something went wrong with that request. Please try again.