From 08e7369e5081dfda13c02b5766c29a169cf642da Mon Sep 17 00:00:00 2001 From: Yohanna Lisnichuk Date: Thu, 25 Feb 2021 17:45:06 -0300 Subject: [PATCH 1/3] docs: add middlewares documentation Signed-off-by: Yohanna Lisnichuk --- docs/contributing/index.rst | 6 ++++++ docs/contributing/middlewares.rst | 6 ++++++ 2 files changed, 12 insertions(+) create mode 100644 docs/contributing/middlewares.rst diff --git a/docs/contributing/index.rst b/docs/contributing/index.rst index 65dfb7d8..1f0bfacc 100644 --- a/docs/contributing/index.rst +++ b/docs/contributing/index.rst @@ -171,6 +171,11 @@ The Scrapy framework is very flexible. To maintain a good separation of concerns - Connect signals, typically `item signals `__ and `spider signals `__ - Raise a `NotConfigured `__ exception in its `from_crawler `__ method, if a required `setting `__ isn't set +- A downloader middleware's responsibility is to process requests, before they are sent to the internet, and responses, before they are processed by the Spiders. It should only: + - Yield a request + - Return a Deferred + - Yield items + When setting a custom `Request.meta key `__, check that the attribute name isn't `already in use `__ by Scrapy. Update requirements @@ -193,3 +198,4 @@ API reference extensions.rst util.rst exceptions.rst + middlewares.rst diff --git a/docs/contributing/middlewares.rst b/docs/contributing/middlewares.rst new file mode 100644 index 00000000..353739ad --- /dev/null +++ b/docs/contributing/middlewares.rst @@ -0,0 +1,6 @@ +Middlewares +=========== + +.. automodule:: kingfisher_scrapy.middlewares + :members: + :undoc-members: From 6f69fbafc168b467405e8c7b8c859bc61155f977 Mon Sep 17 00:00:00 2001 From: Yohanna Lisnichuk Date: Wed, 3 Mar 2021 09:57:41 -0300 Subject: [PATCH 2/3] Update docs/contributing/index.rst Co-authored-by: James McKinney <26463+jpmckinney@users.noreply.github.com> --- docs/contributing/index.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/contributing/index.rst b/docs/contributing/index.rst index 1f0bfacc..eb52463c 100644 --- a/docs/contributing/index.rst +++ b/docs/contributing/index.rst @@ -171,7 +171,7 @@ The Scrapy framework is very flexible. To maintain a good separation of concerns - Connect signals, typically `item signals `__ and `spider signals `__ - Raise a `NotConfigured `__ exception in its `from_crawler `__ method, if a required `setting `__ isn't set -- A downloader middleware's responsibility is to process requests, before they are sent to the internet, and responses, before they are processed by the Spiders. It should only: +- A downloader middleware's responsibility is to process requests, before they are sent to the internet, and responses, before they are processed by the spider. It should only: - Yield a request - Return a Deferred - Yield items From 3dfffeed8958d9361436d9b75923c2c36208022e Mon Sep 17 00:00:00 2001 From: Yohanna Lisnichuk Date: Wed, 3 Mar 2021 10:25:37 -0300 Subject: [PATCH 3/3] docs: add examples of existing middlewares Signed-off-by: Yohanna Lisnichuk --- docs/contributing/index.rst | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/docs/contributing/index.rst b/docs/contributing/index.rst index f23fe07d..72f3eeda 100644 --- a/docs/contributing/index.rst +++ b/docs/contributing/index.rst @@ -162,6 +162,12 @@ The Scrapy framework is very flexible. To maintain a good separation of concerns - Raise a :class:`~kingfisher_scrapy.exceptions.AccessTokenError` exception in a request's callback, if the maximum number of attempts to retrieve an access token is reached - Raise any other exception, to be caught by a `spider_error `__ handler in an extension +- A downloader middleware's responsibility is to process requests, before they are sent to the internet, and responses, before they are processed by the spider. It should only: + + - Yield a request, for example :class:`~kingfisher_scrapy.middlewares.ParaguayAuthMiddleware` + - Return a Deferred, for example :class:`~kingfisher_scrapy.middlewares.DelayedRequestMiddleware` + - Yield items, for example :class:`~kingfisher_scrapy.middlewares.AddPackageMiddleware` + - An item pipeline's responsibility is to clean, validate, filter, modify or substitute items. It should only: - Return an item @@ -173,11 +179,6 @@ The Scrapy framework is very flexible. To maintain a good separation of concerns - Connect signals, typically `item signals `__ and `spider signals `__ - Raise a `NotConfigured `__ exception in its `from_crawler `__ method, if a required `setting `__ isn't set -- A downloader middleware's responsibility is to process requests, before they are sent to the internet, and responses, before they are processed by the spider. It should only: - - Yield a request - - Return a Deferred - - Yield items - When setting a custom `Request.meta key `__, check that the attribute name isn't `already in use `__ by Scrapy. Update requirements