Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

crawlall: Optionally pass specific spiders as arguments #646

Merged
merged 2 commits into from
Mar 2, 2021
Merged

crawlall: Optionally pass specific spiders as arguments #646

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f667314
crawlall: Optionally pass specific spiders as arguments
jpmckinney Feb 27, 2021
49abf85
docs: cli: crawlall: Document spider argument
jpmckinney Mar 2, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/cli.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,7 @@ Runs all spiders.

- ``--dry-run``: Runs the spiders without writing any files. It stops after collecting one file or file item from each spider. This can be used to test whether any spiders are broken. Add the ``--logfile debug.log`` option to write the output to a log file for easier review.
- ``--sample=NUM``: The number of files to write. This can be used to collect a sample from each spider.
- ``spider``: Run specific spiders. Omit to run all spiders.

.. code-block:: bash

Expand Down
5 changes: 4 additions & 1 deletion kingfisher_scrapy/commands/crawlall.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,9 @@ def yield_nothing(*args, **kwargs):


class CrawlAll(ScrapyCommand):
def syntax(self):
return '[options] [spider ...]'

def short_desc(self):
return 'Run all spiders'

Expand Down Expand Up @@ -50,7 +53,7 @@ def run(self, args, opts):
self.settings.set('EXTENSIONS', extensions)

for spider_name in self.crawler_process.spider_loader.list():
if spider_name not in EXCEPTIONS:
if not args and spider_name not in EXCEPTIONS or spider_name in args:
spidercls = self.crawler_process.spider_loader.load(spider_name)
self.crawler_process.crawl(spidercls, **kwargs)

Expand Down