Skip to content

Commit

Permalink
Update cluster-setup docs
Browse files Browse the repository at this point in the history
  • Loading branch information
guillermoap committed Nov 14, 2018
1 parent 0c8c58b commit 037c3b4
Showing 1 changed file with 3 additions and 3 deletions.
6 changes: 3 additions & 3 deletions docs/source/topics/cluster-setup.rst
Expand Up @@ -88,7 +88,7 @@ a common modules and import settings from it in component's modules.
from __future__ import absolute_import
from .worker import *

CRAWLING_STRATEGY = '' # path to the crawling strategy class
STRATEGY = '' # path to the crawling strategy class
LOGGING_CONFIG='logging-sw.conf' # if needed

The logging can be configured according to https://docs.python.org/2/library/logging.config.html see the
Expand Down Expand Up @@ -127,7 +127,7 @@ First, let's start storage worker: ::

# start DB worker only for batch generation
# use single instance for every 10 partitions
$ python -m frontera.worker.db --config [db worker config module] --no-incoming --partitions 0,1
$ python -m frontera.worker.db --config [db worker config module] --no-incoming --partitions 0 1


# Optionally, start next one dedicated to spider log processing.
Expand Down Expand Up @@ -158,4 +158,4 @@ Finally, a single spider per spider feed partition: ::
You should end up with N spider processes running. Also :setting:`SPIDER_PARTITION_ID` can be read from config file.

You're done, crawler should start crawling. Any component can be restarted any time, without major data loss. However,
for pausing its enough to stop batch gen only.
for pausing its enough to stop batch gen only.

0 comments on commit 037c3b4

Please sign in to comment.