[offsite middleware] allow to dynamically add new entries to allowed domains

[pawelmhm]

Currently offsite middleware reads allowed domains from spider attribute on spider opened and uses that to decide whether request should be followed or not.

scrapy/scrapy/spidermiddlewares/offsite.py

Line 58 in 129421c

def spider_opened(self, spider):

I have use case where I'm making some initial request and then need to decide which domains to crawl. So ideally I'd make start_requests and after that set allowed_domains.

Does it make sense to add some way to add allowed domains dynamically? E.g. I could set something like this in spider.

 self.add_allowed_domains('http://foo.com')

and after making this call spider will not follow foo.com.

[Gallaecio]

Not sure about adding a method to Spider, but we could add it to the middleware, which is now easier to reach with

scrapy/scrapy/crawler.py

Line 215 in 1c9d308

def get_spider_middleware(self, cls):

[Gallaecio]

I wonder if we could use Request.meta, e.g. allow_domain=True, and have the middleware pop that key and extend the allowed domains based on the domain of the request URL.