Fix Spider.custom_settings

[kmike]

I noticed that a CloseSpider extension doesn't get activated when CLOSESPIDER_PAGECOUNT option is set in spider's custom_settings. Other extensions (AutoThrottle, LogStats, etc) are also not activated.

This happens because extensions check settings in from_crawler methods, and crawler.settings doesn't contain spider settings yet because spidercls.update_settings call was moved to a later stage by #1128.

[curita]

Makes sense, +1 to merge. I placed wrongly the settings update and freeze in 6f9265b. It should be safe to move the self.settings.freeze() line too, settings were frozen at this point because the extensions shouldn't update them.

[kmike]

@curita I kept settings.freeze() at the bottom because of your comment at #1128:

Settings given to each Crawler will be frozen after the extensions initialization. This allows the extensions to update the settings. That's something we want to implement for the Simplified Addons Idea for this GSoC (issue #591).

Is it no longer needed?

[curita]

Oh, forgot about that comment, I was thinking it's probably better to deal with that once the addons support is implemented. I guess keeping the freeze call where it is is less restrictive now that the change it's already implemented. +1 to merge this pull request as it is then.

[chekunkov]

@kmike @curita I want to ask you a question. Why extensions, middlewares and pipelines are initialized during Crawler.__init__, while spider and engine objects are initialized only during Crawler.crawl? Does it work like that because we want to preserve an ability to run multiple spiders via the same set of middlewares?

I'm asking this because sometimes I feel like I want to change some crawl settings after spider initialization and initialize middlewares only after that. For example I got request from customer to make it possible to set CLOSESPIDER_TIMEOUT based on passed spider argument. Due to CloseSpider implementation to support this I need to override it, disable default and set custom extension in settings. If initialization order was

'spider init' -> 'update settings' -> 'settings.freeze' -> 'middleware init' 

that task would be as easy as set CLOSESPIDER_TIMEOUT in custom_settings

[chekunkov]

@dangra please check comment above, maybe you can explain what design decision is behind current initialization order? I forgot to mention you in the comment, and you're obviously the person who knows almost everything about Scrapy.

[nramirezuy]

@chekunkov -s doesn't works for you?
ex: scrapy crawl <spider> -s CLOSESPIDER_TIMEOUT=900

[chekunkov]

@nramirezuy -s does work in command line, but doesn't work on dash, API doesn't provide a way to set custom settings during schedule and customer wants to be able to set this setting per crawl.

[chekunkov]

Also it could happen that spider has some logic to decide whether or not this setting should be set based on spider arguments, -s doesn't help in this case

[dangra]

@chekunkov: I think you have a point, the order of components initialization is not clear but it is better to propose changing it in a new ticket. Scrapy 1.0 is pretty much ready, so we can work on your problem for 1.1.

[chekunkov]

Opened a new ticket #1305

[kmike]

@chekunkov 👍