public Crawler.create_crawler method #1528
Merged
Conversation
Current coverage is
|
scrapy/utils/test.py
Outdated
| @@ -30,7 +30,7 @@ def get_crawler(spidercls=None, settings_dict=None): | |||
| from scrapy.spiders import Spider | |||
|
|
|||
| runner = CrawlerRunner(Settings(settings_dict)) | |||
curita
Oct 30, 2015
Member
This is unrelated to these changes, but now that I see it we could pass settings_dict as it is to CrawlerRunner, we don't need to wrap it here into a Settings instance.
This is unrelated to these changes, but now that I see it we could pass settings_dict as it is to CrawlerRunner, we don't need to wrap it here into a Settings instance.
* use CrawlerRunner.create_crawler instead of get_crawler helper in test_crawl; * add a test for loading spiders by name; * add a test for passing Crawler objects instead of Spider objects; * add a test for CrawlerRunner.join
|
+1 to merge, but it needs to be rebased. |
curita
added a commit
that referenced
this pull request
Oct 30, 2015
public Crawler.create_crawler method
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
CrawlerRunner.crawl returns a Deferred, but in all cases I used this method in addition to waiting for crawl to finish it was necessary to connect signal handlers, and you need Crawler instance for that. So
was not enough, I wanted
This means it was necessary to copy-paste the same code
in all CrawlerRunner subclasses / wrappers, and using a private
_create_crawlermethod. What about adding a public method for it? As a bonus point, we won't be using a private method inscrapy.utils.test.get_crawler.