Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make it possible to debounce requests #305

Open
barraponto opened this issue May 21, 2013 · 2 comments
Open

Make it possible to debounce requests #305

barraponto opened this issue May 21, 2013 · 2 comments

Comments

@barraponto
Copy link
Contributor

@barraponto barraponto commented May 21, 2013

I just read Ben Alman on jQuery's Throttle and Debounce and wondered: can DebounceRequest be added to Scrapy?

The use case: recently I've been scraping a website where I wanted to gather Facebook Likes per URL. Now that comes cheap using Facebook API, particularly the Facebook Query Language. The problem is that FB will eventually stop answering my request, on a undocumented rate limit. But what if I could define a way for those requests to be joined and called only after a certain while, asking for all the parameters in a user-defined way? I'd expect the callback to be called just once, too (or maybe several times but with the full response).

@nyov
Copy link
Contributor

@nyov nyov commented Apr 5, 2015

I don't actually understand the facebook related parts here.
Were you asking for debouncing requests to same URLs instead of dupe-filtering?
Or would you mind explaining the use-case again?

@barraponto
Copy link
Contributor Author

@barraponto barraponto commented Jun 5, 2015

Let's say my CustomSpider yields DebouncedRequests with a particular parameter, like http://domain.com/getdata.xml?query=1. But instead of firing it immediately, I set rules for it to wait at least 10 requests (or maybe wait for at least 10 seconds) and join the requests on a single request to http://domain.com/getdata.xml?query=1,2,3,4,5,6,7,8,9,10.

I think this should already be possible in a middleware.

@cathalgarvey cathalgarvey added the stale label Feb 20, 2018
lucywang000 pushed a commit to lucywang000/scrapy that referenced this issue Feb 24, 2019
Do not remove existing loggers on configuration
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants
You can’t perform that action at this time.