Add note that "allowed_domains" should be a list of domains, not URLs

[redapple]

(just logging the issue before I forget)

It may seem obvious by the name of the attribute that allowed_domains is about domain names, but it's not uncommon for scrapy users to make the mistake of doing allowed_domains = ['http://www.example.com']

I believe it is worth adding a note in http://doc.scrapy.org/en/latest/topics/spiders.html?#scrapy.spiders.Spider.allowed_domains

[eliasdorneles]

What if instead of simply documenting, Scrapy detect this case and issues a warning?

Even better, it could extract the domain from the URL and use that, while issuing a warning like:

logging.warn("allowed_domains accepts only domains, not URLs. Using allowed_domains = %r" % effective_allowed_domains)

[redapple]

+1 to issue a warning.

I'm less sure about inferring domain, for example for http://www.example.com, should it infer example.com or www.example.com?

[eliasdorneles]

I think the user expectation would be www.example.com.

I think it would be reasonable to use whatever urlparse parses as netloc:

>>> import urlparse
>>> urlparse.urlparse('http://www.example.com')
ParseResult(scheme='http', netloc='www.example.com', path='', params='', query='', fragment='')

[redapple]

Are you ok with having the new behavior as a seperate issue?

[eliasdorneles]

Sure, inferring a domain from an URL in allowed_domains looks like a separate feature indeed.

But I think warning users about URLs detected in allowed_domains and warning users about it belongs to the same issue as documenting (this one).
I mean, it isn't great to point users to docs when we can give them instant feedback.

Makes sense?

[redapple]

It does

[jlong49]

May I have this issue?