GSoC Per-spider settings

[curita]

Per-spider settings task of the 2014 GSoC, implemented on top of the #816 pull request.

This changes are based on the SEP#19.

[curita]

@dangra I've fixed both issues and rebased the new changes in the api-cleanup pull request on this branch.

[curita]

@dangra I've fixed both issues and rebased the new changes in the api-cleanup pull request on this branch.

[dangra]

sorry for been late, but I think RuntimeError is better than TypeError in this case, the latter is much more common mistake. @kmike what do you think?

[dangra]

sorry for been late, but I think RuntimeError is better than TypeError in this case, the latter is much more common mistake. @kmike what do you think?

[kmike]

I suggested TypeError because this is what immutable containers raise, e.g.

In [1]: (3,2)[0] = 1
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-1-cab3fa073914> in <module>()
----> 1 (3,2)[0] = 1

TypeError: 'tuple' object does not support item assignment

[dangra]

ok, fair point :)

…

On Thu, Aug 14, 2014 at 2:34 PM, Mikhail Korobov ***@***.*** wrote: I suggested TypeError because this is what immutable containers raise, e.g. ## In [1]: (3,2)[0] = 1 TypeError Traceback (most recent call last) <ipython-input-1-cab3fa073914> in <module>() ----> 1 (3,2)[0] = 1 TypeError: 'tuple' object does not support item assignment — Reply to this email directly or view it on GitHub 70f2010#commitcomment-7393006 .

[kmike]

It seems that CrawlerRunner.spiders is not documented. I'm not sure I like that CrawlerSpider knows about SpiderManager (and it can be used without it, as shown in your new example).

[dangra]

Merged by 5daa147

[dangra]

Merged by 5daa147

[raintan]

This comment should be removed or changed to: # 'testspider' is the name...

[raintan]

This comment should be removed or changed to: # 'testspider' is the name...

[curita]

Not sure if I'm misunderstanding what you're saying @raintan, but testspiders is the name of the project (which you can found here: https://github.com/scrapinghub/testspiders, should be linked in the docs as well), and followall is the name of the spider inside that project that we're using (You can check the code in the line below).

[raintan]

In the latest version of the docs, followall isn't part of the code example anymore.
see: http://doc.scrapy.org/en/1.0/topics/practices.html right after "What follows is a working example of how to do that, using the testspiders project as example."
(I'm new here, so I'm not sure if this was the proper place to put this comment.)

[curita]

Oh, I see what you meant, thanks for the notice!

This issue seems to be fixed already (that's why these docs are correct), but the latest available version of scrapy docs is this one: http://doc.scrapy.org/en/master/topics/practices.html (I know, rather confusing that is not latest). We'll see to backport the fix to 1.0 accordantly.

Generally opening an issue for any kind of bugs is the way to go, it's more flexible and github offers more features for them (can be referenced easier, we can modify their status to track the work done on them, milestones can be assigned to each of them, etc), so feel free to open an issue for your next bug report :)