[MRG+1] Enable genspider command outside project folder

[stummjr]

This PR enables the genspider CLI command even when the working dir is not a scrapy project.

The rationale behind is that some users (including me) are used to create standalone spiders and run it with the runspider command, because it's a quick and convenient way to fire up simple spiders. Having genspider available would make it even quicker.

[codecov-io]

Current coverage is 83.32%

Merging #2052 into master will decrease coverage by <.01%

@@             master      #2052   diff @@
==========================================
  Files           161        161          
  Lines          8678       8682     +4   
  Methods           0          0          
  Messages          0          0          
  Branches       1272       1274     +2   
==========================================
+ Hits           7231       7234     +3   
  Misses         1196       1196          
- Partials        251        252     +1   

Powered by Codecov. Last updated by 759a555...081595a

[eliasdorneles]

+1, this is cool!

Only thing missing now is to update docs. :)

genspider is listed as a project-only command here: http://doc.scrapy.org/en/latest/topics/commands.html#available-tool-commands

The short description for genspider should also be updated, and it would be nice to mention the meaning of the arguments when using standalon (like, how name is now used for the spider file name).

[stummjr]

This PR fails when setting the template for crawl, xmlfeed or csvfeed, because those templates include code to import the items module.

I'm going to have a look on how to fix it and then I update the PR.

[stummjr]

I was looking at how to make this work for the 'crawl', 'xmlfeed' and 'csvfeed' templates. The issue is that those templates import the <classname>Item module and we don't have such module when generating a standalone spider.

We could solve that by removing the import and uses of the Item class in the spider code, as it is in the 'basic' template. The crawl template would become:

import scrapy
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule


class $classname(CrawlSpider):
    name = '$name'
    allowed_domains = ['$domain']
    start_urls = ['http://www.$domain/']

    rules = (
        Rule(LinkExtractor(allow=r'Items/'), callback='parse_item', follow=True),
    )

    def parse_item(self, response):
        pass

Or, we could employ a template engine, such as jinja2, but it looks like an overkill.

Thoughts?

[eliasdorneles]

+1 on changing the crawl template.
Since we've added support for dict items, we should probably not assume user is declaring items.

[eliasdorneles]

@stummjr looks good!
Can you update the docs?
Will be ready to merge after that. :)

[eliasdorneles]

The build is failing because of unrelated coverage error.
I bet it's the new version of pytest-cov released today.
Try updating test/requirements.txt with a version specifier for pytest-cov<=2.3.0 to see if build passes.

[eliasdorneles]

sorry, I should've said pytest-cov!=2.3.0!

[eliasdorneles]

Thanks @stummjr !