Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[MRG+1] Better explain middleware orders and processing directions #2329

Merged
merged 3 commits into from Oct 20, 2016
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
15 changes: 8 additions & 7 deletions docs/topics/architecture.rst
Expand Up @@ -41,25 +41,26 @@ this:

4. The :ref:`Engine <component-engine>` sends the Requests to the
:ref:`Downloader <component-downloader>`, passing through the
:ref:`Downloader Middleware <component-downloader-middleware>`
(requests direction).
:ref:`Downloader Middlewares <component-downloader-middleware>` (see
:meth:`~scrapy.downloadermiddlewares.DownloaderMiddleware.process_request`).

5. Once the page finishes downloading the
:ref:`Downloader <component-downloader>` generates a Response (with
that page) and sends it to the Engine, passing through the
:ref:`Downloader Middleware <component-downloader-middleware>`
(response direction).
:ref:`Downloader Middlewares <component-downloader-middleware>` (see
:meth:`~scrapy.downloadermiddlewares.DownloaderMiddleware.process_response`).

6. The :ref:`Engine <component-engine>` receives the Response from the
:ref:`Downloader <component-downloader>` and sends it to the
:ref:`Spider <component-spiders>` for processing, passing
through the :ref:`Spider Middleware <component-spider-middleware>`
(input direction).
through the :ref:`Spider Middleware <component-spider-middleware>` (see
:meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_input`).

7. The :ref:`Spider <component-spiders>` processes the Response and returns
scraped items and new Requests (to follow) to the
:ref:`Engine <component-engine>`, passing through the
:ref:`Spider Middleware <component-spider-middleware>` (output direction).
:ref:`Spider Middleware <component-spider-middleware>` (see
:meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_output`).

8. The :ref:`Engine <component-engine>` sends processed items to
:ref:`Item Pipelines <component-pipelines>`, then send processed Requests to
Expand Down
6 changes: 5 additions & 1 deletion docs/topics/downloader-middleware.rst
Expand Up @@ -27,7 +27,11 @@ The :setting:`DOWNLOADER_MIDDLEWARES` setting is merged with the
:setting:`DOWNLOADER_MIDDLEWARES_BASE` setting defined in Scrapy (and not meant
to be overridden) and then sorted by order to get the final sorted list of
enabled middlewares: the first middleware is the one closer to the engine and
the last is the one closer to the downloader.
the last is the one closer to the downloader. In other words,
the :meth:`~scrapy.downloadermiddlewares.DownloaderMiddleware.process_request`
method of each middleware will be invoked in increasing
middleware order (100, 200, 300, ...) and the :meth:`~scrapy.downloadermiddlewares.DownloaderMiddleware.process_response` method
of each middleware will be invoked in decreasing order.

To decide which order to assign to your middleware see the
:setting:`DOWNLOADER_MIDDLEWARES_BASE` setting and pick a value according to
Expand Down
7 changes: 6 additions & 1 deletion docs/topics/spider-middleware.rst
Expand Up @@ -28,7 +28,12 @@ The :setting:`SPIDER_MIDDLEWARES` setting is merged with the
:setting:`SPIDER_MIDDLEWARES_BASE` setting defined in Scrapy (and not meant to
be overridden) and then sorted by order to get the final sorted list of enabled
middlewares: the first middleware is the one closer to the engine and the last
is the one closer to the spider.
is the one closer to the spider. In other words,
the :meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_input`
method of each middleware will be invoked in increasing
middleware order (100, 200, 300, ...), and the
:meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_output` method
of each middleware will be invoked in decreasing order.

To decide which order to assign to your middleware see the
:setting:`SPIDER_MIDDLEWARES_BASE` setting and pick a value according to where
Expand Down