scrapy.telnet
Scrapy comes with a built-in telnet console for inspecting and controlling a Scrapy running process. The telnet console is just a regular python shell running inside the Scrapy process, so you can do literally anything from it.
The telnet console is a built-in Scrapy extension
<topics-extensions-ref>
which comes enabled by default, but you can also disable it if you want. For more information about the extension itself see topics-extensions-ref-telnetconsole
.
none
The telnet console listens in the TCP port defined in the TELNETCONSOLE_PORT
setting, which defaults to 6023
. To access the console you need to type:
telnet localhost 6023
>>>
You need the telnet program which comes installed by default in Windows, and most Linux distros.
The telnet console is like a regular Python shell running inside the Scrapy process, so you can do anything from it including importing new modules, etc.
However, the telnet console comes with some default variables defined for convenience:
Shortcut | Description |
---|---|
crawler |
the Scrapy Crawler (scrapy.crawler.Crawler object) |
engine |
Crawler.engine attribute |
spider |
the active spider |
slot |
the engine slot |
extensions |
the Extension Manager (Crawler.extensions attribute) |
stats |
the Stats Collector (Crawler.stats attribute) |
settings |
the Scrapy settings object (Crawler.settings attribute) |
est |
print a report of the engine status |
prefs |
for memory debugging (see topics-leaks ) |
p |
a shortcut to the pprint.pprint function |
hpy |
for memory debugging (see topics-leaks ) |
Here are some example tasks you can do with the telnet console:
You can use the est()
method of the Scrapy engine to quickly show its state using the telnet console:
telnet localhost 6023
>>> est()
Execution engine status
time()-engine.start_time : 8.62972998619
engine.has_capacity() : False
len(engine.downloader.active) : 16
engine.scraper.is_idle() : False
engine.spider.name : followall
engine.spider_is_idle(engine.spider) : False
engine.slot.closing : False
len(engine.slot.inprogress) : 16
len(engine.slot.scheduler.dqs or []) : 0
len(engine.slot.scheduler.mqs) : 92
len(engine.scraper.slot.queue) : 0
len(engine.scraper.slot.active) : 0
engine.scraper.slot.active_size : 0
engine.scraper.slot.itemproc_size : 0
engine.scraper.slot.needs_backout() : False
To pause:
telnet localhost 6023
>>> engine.pause()
>>>
To resume:
telnet localhost 6023
>>> engine.unpause()
>>>
To stop:
telnet localhost 6023
>>> engine.stop()
Connection closed by foreign host.
update_telnet_vars
update_telnet_vars(telnet_vars)
Sent just before the telnet console is opened. You can hook up to this signal to add, remove or update the variables that will be available in the telnet local namespace. In order to do that, you need to update the telnet_vars
dict in your handler.
- param telnet_vars
the dict of telnet variables
- type telnet_vars
dict
These are the settings that control the telnet console's behaviour:
TELNETCONSOLE_PORT
Default: [6023, 6073]
The port range to use for the telnet console. If set to None
or 0
, a dynamically assigned port is used.
TELNETCONSOLE_HOST
Default: '0.0.0.0'
The interface the telnet console should listen on