Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

deprecate Spider.make_requests_from_url #1495

kmike opened this issue Sep 13, 2015 · 2 comments

deprecate Spider.make_requests_from_url #1495

kmike opened this issue Sep 13, 2015 · 2 comments


Copy link

@kmike kmike commented Sep 13, 2015


Spider.make_requests_from_url is a shortcut for a single use case: sending initial requests from a spider with start_urls attribute. It saves a single line of code (for url in start_urls) in one specific use case, but yet we have to document it, users have to read about it and understand what it is for.

The name suggests it is more general - but we can't yield Link objects and get requests created by make_requests_from_url, it does nothing for CrawlSpider (I'd expect URLs to be passed through this method), and it is ignored for SitemapSpider. This is inconsistent.

What about deprecating make_requests_from_url and removing it from docs? IMHO it will make API simpler, this hook does nothing useful now, and if we ever want a general 'process URL and get a Request' method we'd have to use another name for backwards compatibility anyways.

Copy link

@nramirezuy nramirezuy commented Sep 14, 2015

I agree; this method isn't that useful. Also now you even have a shorter import for Request if you wanna be lazy (:

Copy link

@chekunkov chekunkov commented Oct 3, 2015


Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

3 participants
You can’t perform that action at this time.