LWP::Simple does not pull down entire page. #10

Closed
ronaldxs opened this Issue Jul 28, 2012 · 7 comments

Comments

Projects
None yet
3 participants
Contributor

ronaldxs commented Jul 28, 2012

It looks from the code like it is just pulling down one packet. For example if you

perl6 -MLWP::Simple -e 'say LWP::Simple.get("http://perlcabal.org/syn/S02.html");'

you just get back the top of the page. I am tempted to add this as test case with a match on something at the bottom of the page to verify we pulled the whole thing.

Ron

Owner

cosimo commented Jul 28, 2012

It looks from the code like it is just pulling down one packet. For
example if you

perl6 -MLWP::Simple -e 'say
LWP::Simple.get("http://perlcabal.org/syn/S02.html");'

you just get back the top of the page. I am tempted to add this as test
case with a match on something at the bottom of the page to verify we
pulled the whole thing.

I think this is a problem with chunked encoding...
But in any case, even if it's not, sure it's a good idea
to add a test case.

Cosimo

Contributor

azawawi commented Oct 21, 2012

I encountered this problem while trying to use LWP::Simple with https://github.com/azawawi/farabi6/. Any chance we could fix this soon?

Contributor

ronaldxs commented Oct 21, 2012

Actually, I overlooked closing this problem when I fixed it many weeks ago. If you are using a current (September) Rakudo Star you should not have it. If you continue to have the problem please check your versions of Rakudo Star/Rakudo as well as LWP::Simple and post the software versions along with the page you are trying to pull with LWP::Simple and any relevant error messages here. Thanks.

Contributor

azawawi commented Oct 21, 2012

Thanks for the quick response. Try this on rakudo 2012.09.

perl6 -MLWP::Simple -e 'say LWP::Simple.get("http://rosettacode.org");'
Contributor

ronaldxs commented Oct 24, 2012

Thank you for posting the example. The page resides on a web server that serves pages in a way not properly handled by existing code and fixes. I have created a fix for this problem, at least with current Rakudo Star, and pushed it to my fork of the project ronaldxs / perl6-lwp-simple. When my fork is merged into the main version of the project I will post a comment here and you should be able to install the fix with Panda.

Cosimo - if you see this could you please follow through on my pull request? Thanks.

Owner

cosimo commented Oct 24, 2012

On Wed, 24 Oct 2012 15:11:51 +0200, ronaldxs notifications@github.com
wrote:

Cosimo - if you see this could you please follow through on my pull
request? Thanks.

Done now. Thanks!

Cosimo

Owner

cosimo commented Oct 22, 2015

We have a few tests about this issue now and they all pass as of current master.

@cosimo cosimo closed this Oct 22, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment