-
-
Notifications
You must be signed in to change notification settings - Fork 357
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Possible recrawl bugs in v2.1 #59
Comments
Hi @halfer It looks like you forgot about the different versions of your tutorial ;) The public function hasBeenCrawled(Url $url, $response, Url $foundOnUrl = null)
{
echo sprintf("Crawled URL: %s\n", $url);
} You'll see that every page including every different version is being crawled as intended!
Hopefully this clears some things up for you. |
Ha ha, thanks @AlexVanderbist - oops! I did completely forget about that. That presents an interesting question then - v1.3 does not behave in the same way, so does that not take query strings into account when differentiating URLs? |
I initially thought this might be a duplicate of issue #3. I have a simple site I am scraping (just for test purposes, it's my own site) and I am seeing some unusual behaviour in terms of URLs being scraped more than once. I am expecting a successful crawl to record about 10 pages, with one crawl per page.
In the above-mentioned issue, attention was drawn to the method
hasAlreadyCrawled()
, which does not seem to be present in the 2.x releases. I don't know if that is relevant.hasBeenCrawled
in theCrawlObserver
many times for each URL.shouldCrawl()
, then the system seems to run the CPU at 100% for many minutes, and needs cancelling via^C
.I think 1.3 is mostly OK, with one surprising gotcha:
shouldCrawl()
is called for all 13, buthasBeenCrawled()
is only called for the root page. I guess this is not a bug, and that I should simply not be trying to keep track of dups on my end.Here is the script I am using with 1.3, you can see I've commented out things that are for the 2.x branch. This should run without any modifications being necessary. I've copied
CrawlInternalUrls
into the script, as this is not included in that release.Here is the script I am using with 2.1, and this is what the output looks like (from
hasBeenCrawled()
):As you can see there's a lot of dups in there. That's version 2.1.2, on Ubuntu, using PHP 7.1.x.
I will fall back to 1.3.x for the time being, but do let me know if you want me to do any testing for you - happy to help if I can.
The text was updated successfully, but these errors were encountered: