Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

deprecate Spider.make_requests_from_url #1495

Closed
kmike opened this issue Sep 13, 2015 · 2 comments
Closed

deprecate Spider.make_requests_from_url #1495

kmike opened this issue Sep 13, 2015 · 2 comments
Labels

Comments

@kmike
Copy link
Member

@kmike kmike commented Sep 13, 2015

Hey,

Spider.make_requests_from_url is a shortcut for a single use case: sending initial requests from a spider with start_urls attribute. It saves a single line of code (for url in start_urls) in one specific use case, but yet we have to document it, users have to read about it and understand what it is for.

The name suggests it is more general - but we can't yield Link objects and get requests created by make_requests_from_url, it does nothing for CrawlSpider (I'd expect URLs to be passed through this method), and it is ignored for SitemapSpider. This is inconsistent.

What about deprecating make_requests_from_url and removing it from docs? IMHO it will make API simpler, this hook does nothing useful now, and if we ever want a general 'process URL and get a Request' method we'd have to use another name for backwards compatibility anyways.

@nramirezuy
Copy link
Contributor

@nramirezuy nramirezuy commented Sep 14, 2015

I agree; this method isn't that useful. Also now you even have a shorter import for Request if you wanna be lazy (:

@chekunkov
Copy link
Contributor

@chekunkov chekunkov commented Oct 3, 2015

+1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

3 participants
You can’t perform that action at this time.