Request throttling

[Ziinc]

This feature would involve rate limiting of requests made by a spider, such as X requests per min.

[oltarasenko]

Request limiting is achieved by the number of workers atm. Could you explain why it does not meet your needs?

[Ziinc]

Yes currently I drop the worker count to one, but it still averages 60~70 requests per minute, which is still a tad too high for my liking. That is over 3600 requests per hour, which would likely be flagged out in anomaly-based firewall systems.

So far I haven't had much issue, but it would be nice to have granular control over the requests/min.

Perhaps tag this as a "nice-to-have"?

[oltarasenko]

Wow. I can't get more than 50 rpm from two workers on the CrawlyUI demo for an example.

I was not expecting it to be a problem, but indeed, it should be fixed.
A worker does a request once per 300 microseconds (https://github.com/oltarasenko/crawly/blob/master/lib/crawly/worker.ex#L11). Usually, the HTTP part is a bottleneck here. However, we can make it configurable. Let me implement this quickly, so you could have it sooner.

Do you need it fast? I can do 0.10.1 for the case tomorrow.

[Ziinc]

It isn't urgent, no need to rush it. Its just something I noticed and was on my mind since we had the discussion in Jan about how requests were fetched #39 (comment) as this type of throttling customization could be achieved through a a "pipeline" module between the fetching and then datastorage portion in the diagram.

For example, I could do things like randomize the throttle rate, or base the throttle rate on some calculation.

But yea, for some reason my machine makes the requests quite quickly.

[oltarasenko]

@Ziinc Actually I don't want to have the flexibility of assigning a given speed to a given worker. It will produce unpredictable results when different workers will have different speeds. So it will be hard to reason why something is faster and something is slower.

Currently, my mind suggests me to hardcode the worker's speed at some value. For all workers. E.g. we can have 5 requests per minute from a worker at max. Or even 1 request per minute from a worker. What do you think?

[oltarasenko]

@Ziinc I have made this: #111

Hopefully, it could improve your case.

[Ziinc]

Many thanks, so adjusting the request rate will be based on # of workers? an interesting approach. Will review the PR