Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

extract CrawlerRunner._crawl method which always expects Crawler #1290

Merged
merged 1 commit into from Jun 26, 2015

Conversation

@kmike
Copy link
Member

@kmike kmike commented Jun 8, 2015

It provides an extension point where crawler instance is available; it should make it easier to write alternative CrawlerRunner.crawl implementations.

My use case: listen to signals from each crawler. As as @chekunkov pointed in #1256 (comment), currently it requires copy-pasting large chunks of code. After this change user can override CrawlerRunner._crawl method and connect signals there.

I don't know if we should make this method public & documented or not.

It provides an extension point where crawler instance is available;
it should make it easier to write alternative CrawlerRunner.crawl
implementations.

See also: #1256
dangra added a commit that referenced this pull request Jun 26, 2015
extract CrawlerRunner._crawl method which always expects Crawler
@dangra dangra merged commit 5bcda9b into master Jun 26, 2015
2 checks passed
2 checks passed
continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details
@dangra dangra deleted the crawler-runner-cleanup branch Jun 26, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

2 participants
You can’t perform that action at this time.