Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Request limit enhancement - limit number of requests per x interval #3920

Closed
dkwolfe4 opened this issue May 22, 2019 · 5 comments

Comments

@dkwolfe4
Copy link

commented May 22, 2019

I'm currently requesting multiple calls to a given host/API endpoint using curl's multiple URL sequences from a command line using

 curl "http://www.somehost.com/v1/api?input={1,2,3,4,5}" -o "#1_file.txt"

The connection to the host is kept open for subsequent requests. The problem is that the API like most APIs these days is request rate limited, and the --limit-rate command does not help as the files returned from the API can be anywhere from <10k to several MB in size. If API calls results in several small files in a row, curl's performance can cause the next API call to be rejected due to rate limits - in this case no more than 2 per second. On the other hand, if I set the limit rate to an artificially low value to handle with these small files, curl's performance on the larger files is lousy.

The situation is not new and as been raised in the past with no resolution that I can find. From the curl mail archives in Sept 2017:

... I can limit the bandwidth usage but not how many requests per x time from a list of urls.... https://curl.haxx.se/mail/archive-2017-09/0018.html

The curl tool master client github wiki page shows the style of script that exists everywhere as a result of this limitation.

#!/bin/sh
for i in 1 .. 100000; do
       curl https://example.com/api?input= i  yadayada
       sleep X seconds
done

I'd like to avoid writing and maintaining such an external script just control curl's request rate. Instead, I'd like to see an option added to curl that limits the number of requests on a per host basis to no more that X requests in a given interval period. The number of requests and the interval should be user settable as the API rate limits vary widely in both request limits and time intervals.

@bagder

This comment has been minimized.

Copy link
Member

commented May 22, 2019

A simpler to use approach would be to simply ask for at least NNN milliseconds between each new request. Then you could for example specify two requests per second with something like (pretending the option is called --after):

curl "https://example.com/api?input=[1-1000]" -d yadayada --after 500

In a command line use setting, setting different options per host name is a bit cumbersome.

@dkwolfe4

This comment has been minimized.

Copy link
Author

commented May 22, 2019

I agree that different options per host name would be difficult... and potentially disruptive. Your proposed approach would fix the most common usage case of calling a single API endpoint where the API rate limits are known ahead of time. Hopefully this simpler solution would be easier to implement.

@dkwolfe4 dkwolfe4 changed the title Request limit enhancement - limit number of requests per host for x interval Request limit enhancement - limit number of requests per x interval May 22, 2019

@bagder bagder closed this in 8c88e8e Jun 3, 2019

@dkwolfe4

This comment has been minimized.

Copy link
Author

commented Jun 3, 2019

And the reason for the closure?

@bagder

This comment has been minimized.

Copy link
Member

commented Jun 3, 2019

We use this tracker for bugs and issues only, we put ideas to work on in the future in the TODO document. We basically drown in good ideas so they don't do much use in our tracker.

This idea was added to the TODO in the commit that closed this issue.

@dkwolfe4

This comment has been minimized.

Copy link
Author

commented Jun 3, 2019

Thank you for the explanation. I was unaware there was a TODO list for future enhancements.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.