curl: add --rate to set max request rate per time unit

[bagder]

--rate "12/m" - for 12 per minute or
--rate "5/h" - for 5 per hour

This option is only for serial transfers, when curl is told to do more than one request.

• man page section added

[jay]

what's with the slash? how about 12s 5m 1h. or if you're worried 5m will be confused with limit-rate then how about --request-rate 5min

[jzakrzewski]

The slash looks better IMHO. Consider what happens when you read it: "12s" is "12 seconds", where "12/s" is "12 per second" kinda without thinking. So without the slash "12s" would be like "once every 12 seconds" to me, which is not what the option does, right?

[bagder]

Yes, my thinking was exactly what @jzakrzewski said here.

[jay]

So without the slash "12s" would be like "once every 12 seconds" to me, which is not what the option does, right?

Yes you're right. Though I still like --request-rate. Why is the option limited to serial transfers?

[bagder]

Though I still like --request-rate.

I was considering that, but then I wasn't sure if request is really the best term when speaking other protocols than HTTP as well - it would perhaps be better with --transfer-rate but that also didn't feel appropriate so I got a little lost. 😁

Why is the option limited to serial transfers?

How would it work for parallel? I couldn't think of how it should work so I left that out until we can figure that out.

[Whissi]

Hi!

Your new parameter still requires manual interaction, i.e. user has to know and set proper rate limiting on their own.

Any chance to implement support for X-Ratelimit-* headers as described in https://tools.ietf.org/id/draft-polli-ratelimit-headers-02.html which I think Github, Twitter and some others have already implemented?

[bagder]

Correct (and note that you link to an outdated version of the draft). I'm going for simplicity to start with.

If we want curl to adapt to what the response headers' say, then I think we should make that an explicit instruction for the --rate option. Maybe --rate as-indiciated or something, and we probably even need to have the option as "use this rate or whatever rate the site might say".

Something to keep in mind when thinking of letting it adapt to what the server says: as this rate frequency feature is implemented now, it works independently of which servers curl speaks to. It means that you could do curl $host1 $host2 $host3 and they could very well each return a different set of ratelimit headers...