Skip to content

Latest commit

 

History

History
115 lines (76 loc) · 3.15 KB

exceptions.rst

File metadata and controls

115 lines (76 loc) · 3.15 KB
Raw

Exceptions

scrapy.exceptions

Built-in Exceptions reference

Here's a list of all exceptions included in Scrapy and their usage.

CloseSpider

CloseSpider(reason='cancelled')

This exception can be raised from a spider callback to request the spider to be closed/stopped. Supported arguments:

param reason

the reason for closing

type reason

str

For example:

def parse_page(self, response):
    if 'Bandwidth exceeded' in response.body:
        raise CloseSpider('bandwidth_exceeded')

DontCloseSpider

DontCloseSpider

This exception can be raised in a spider_idle signal handler to prevent the spider from being closed.

DropItem

DropItem

The exception that must be raised by item pipeline stages to stop processing an Item. For more information see topics-item-pipeline.

IgnoreRequest

IgnoreRequest

This exception can be raised by the Scheduler or any downloader middleware to indicate that the request should be ignored.

NotConfigured

NotConfigured

This exception can be raised by some components to indicate that they will remain disabled. Those components include:

  • Extensions
  • Item pipelines
  • Downloader middlewares
  • Spider middlewares

The exception must be raised in the component's __init__ method.

NotSupported

NotSupported

This exception is raised to indicate an unsupported feature.

StopDownload

2.2

StopDownload(fail=True)

Raised from a ~scrapy.signals.bytes_received or ~scrapy.signals.headers_received signal handler to indicate that no further bytes should be downloaded for a response.

The fail boolean parameter controls which method will handle the resulting response:

  • If fail=True (default), the request errback is called. The response object is available as the response attribute of the StopDownload exception, which is in turn stored as the value attribute of the received ~twisted.python.failure.Failure object. This means that in an errback defined as def errback(self, failure), the response can be accessed though failure.value.response.
  • If fail=False, the request callback is called instead.

In both cases, the response could have its body truncated: the body contains all bytes received up until the exception is raised, including the bytes received in the signal handler that raises the exception. Also, the response object is marked with "download_stopped" in its Response.flags attribute.

Note

fail is a keyword-only parameter, i.e. raising StopDownload(False) or StopDownload(True) will raise a TypeError.

See the documentation for the ~scrapy.signals.bytes_received and ~scrapy.signals.headers_received signals and the topics-stop-response-download topic for additional information and examples.