Scrapy extension to control spiders using JSON-RPC
Clone or download
kmike Merge pull request #13 from alosultan/master
Module `scrapy.spider` is deprecated. Used 'six.string_types' instead 'basestring'
Latest commit 15e0b70 Mar 21, 2017



scrapy-jsonrpc is an extension to control a running Scrapy web crawler via JSON-RPC. The service provides access to the main Crawler object via the JSON-RPC 2.0 protocol.


Install scrapy-jsonrpc using pip:

$ pip install scrapy-jsonrpc


First, you need to include the entension to your EXTENSIONS dict in, for example:

    'scrapy_jsonrpc.webservice.WebService': 500,

Then, you need to enable the extension with the JSONRPC_ENABLED setting, set to True.

The web server will listen on a port specified in JSONRPC_PORT (by default, it will try to listen on port 6080), and will log to the file specified in JSONRPC_LOGFILE.

The endpoint for accessing the crawler object is:


Example client

There is a command line tool provided for illustration purposes on how to build a client. You can find it in It supports a few basic commands such as listing the running spiders, etc.


These are the settings that control the web service behaviour:


Default: False

A boolean which specifies if the web service will be enabled (provided its extension is also enabled).


Default: None

A file to use for logging HTTP requests made to the web service. If unset web the log is sent to standard scrapy log.


Default: [6080, 7030]

The port range to use for the web service. If set to None or 0, a dynamically assigned port is used.


Default: ''

The interface the web service should listen on.