Cookiejars exposed

docs/faq.rst

@@ -307,7 +307,7 @@ Does Scrapy manage cookies automatically?
 Yes, Scrapy receives and keeps track of cookies sent by servers, and sends them
 back on subsequent requests, like any regular web browser does.
 
-For more info see :ref:`topics-request-response` and :ref:`cookies-mw`.
+For more info see :ref:`topics-request-response`, :ref:`cookies-mw` and :ref:`cookiejars`.
 
 How can I see the cookies being sent and received from Scrapy?
 --------------------------------------------------------------

docs/topics/downloader-middleware.rst

@@ -293,6 +293,54 @@ Here's an example of a log with :setting:`COOKIES_DEBUG` enabled::
     [...]
 
 
+.. _cookiejars:
+
+Direct access to cookiejars from spider
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In some cases it is required to directly set specific values in an existing
+cookiejar.
+
+.. code-block:: python
+
+    import scrapy
+    from scrapy.crawler import CrawlerProcess
+
+
+    class Quotes(scrapy.Spider):
+        name = "quotes"
+        custom_settings = {"DOWNLOAD_DELAY": 1}
+
+        def start_requests(self):
+            yield scrapy.Request(
+                url="https://quotes.toscrape.com/login", callback=self.login
+            )
+
+        def login(self, response):
+            self.logger.info(self.cookie_jars[None])  # scrapy.http.cookies.CookieJar object
+            self.logger.info(self.cookie_jars[None].jar)  # http.cookiejar object
+
+            locale_cookie = (
+                self.cookie_jars[None]._cookies["quotes.toscrape.com"]["/"].get("session")
+            )
+            locale_cookie.value = locale_cookie.value.upper()
+            self.logger.info(self.cookie_jars[None].jar)
+
+
+    if __name__ == "__main__":
+        p = CrawlerProcess()
+        p.crawl(Quotes)
+        p.start()
+
+Log output::
+
+    2024-02-23 10:51:27 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://quotes.toscrape.com/login> (referer: None)
+    2024-02-23 10:51:27 [quotes] INFO: <scrapy.http.cookies.CookieJar object at 0x00000217DB719B40>
+    2024-02-23 10:51:27 [quotes] INFO: <CookieJar[<Cookie session=eyJjc3JmX3Rva2VuIjoiSnFQQU9GTGt1amZzZ3J3UVdHeGV6WFR2UnBpY0Job1NWS3liWmxhblVISXROREVtQ2RZTSJ9.Zdhqng.8uQzjuvDfOcNJHV7luY5Na6C1N0 for quotes.toscrape.com/>]>
+    2024-02-23 10:51:27 [quotes] INFO: <CookieJar[<Cookie session=EYJJC3JMX3RVA2VUIJOISNFQQU9GTGT1AMZZZ3J3UVDHEGV6WFR2UNBPY0JOB1NWS3LIWMXHBLVISXROREVTQ2RZTSJ9.ZDHQNG.8UQZJUVDFOCNJHV7LUY5NA6C1N0 for quotes.toscrape.com/>]>
+    2024-02-23 10:51:27 [scrapy.core.engine] INFO: Closing spider (finished)
+
+
 DefaultHeadersMiddleware
 ------------------------
 

scrapy/downloadermiddlewares/cookies.py

@@ -16,7 +16,7 @@
 
 from tldextract import TLDExtract
 
-from scrapy import Request, Spider
+from scrapy import Request, Spider, signals
 from scrapy.crawler import Crawler
 from scrapy.exceptions import NotConfigured
 from scrapy.http import Response
@@ -51,7 +51,12 @@ def __init__(self, debug: bool = False):
     def from_crawler(cls, crawler: Crawler) -> Self:
         if not crawler.settings.getbool("COOKIES_ENABLED"):
             raise NotConfigured
-        return cls(crawler.settings.getbool("COOKIES_DEBUG"))
+        o = cls(crawler.settings.getbool("COOKIES_DEBUG"))
+        crawler.signals.connect(o.spider_opened, signal=signals.spider_opened)
+        return o
+
+    def spider_opened(self, spider: Spider) -> None:
+        spider.cookie_jars = self.jars
 
     def _process_cookies(
         self, cookies: Iterable[Cookie], *, jar: CookieJar, request: Request