Genspider command prepends an inconvenient 'www' in start_urls

[stummjr]

Almost every time I use the genspider command, I end up removing the www prefix from the value that this command generates for start_urls.

For example, this command:

$ scrapy genspider quotes quotes.toscrape.com

generates this spider:

class QuotesSpider(scrapy.Spider):
    name = "quotes"
    allowed_domains = ["quotes.toscrape.com"]
    start_urls = (
        'http://www.quotes.toscrape.com/',
    )
    ...

IMHO, there's no need to add www to the start_urls, because it might be more annoying than helpful to people using genspider.

Thoughts on it?

[eliasdorneles]

+1 to remove the www from the templates, that seems unnecessary.

[kmike]

+1

[eLRuLL]

Totally agree with @stummjr , www.domain.com is a different site that domain.com

[ThunderMind2019]

genspider also prepend http:// But when i enter address like https://example.com it becomes http://https://example.com that, when run scrapy crawl throws an error.
What it should do, it should first check the receiving domain than take decision according to the passing domain whether it needs a http:// or nothing.

[kmike]

@ThunderMind2019 would you mind creating a separate ticket for this?

[ThunderMind2019]

ok sure thing.