Skip to content

add delay option to work around rate limits#101

Merged
tlimoncelli merged 1 commit intoStackExchange:masterfrom
rbelnap:delay
Aug 5, 2017
Merged

add delay option to work around rate limits#101
tlimoncelli merged 1 commit intoStackExchange:masterfrom
rbelnap:delay

Conversation

@rbelnap
Copy link
Copy Markdown
Contributor

@rbelnap rbelnap commented Apr 26, 2017

I found this was useful to avoid hitting overzealous rate limits. I'm open to other approaches.

@captncraig
Copy link
Copy Markdown
Contributor

The optimal solution for rate limits IMO would be for each provider to know the limits, and self-throttle across all domains.

In the case of nameheap though, I have not been able to find any reliable hard numbers. What value of -d has been working for you?

@rbelnap
Copy link
Copy Markdown
Contributor Author

rbelnap commented Apr 26, 2017

I've found 5 seconds gets me by - which is a far cry from namecheap's vague "you shouldn't hit this limit" comments. It seems to be roughly 10 requests within 20 seconds will trigger it. I'll take it up with namecheap as well.

I wouldn't mind seeing it as a property of the provider (or set in the DSL perhaps) - I don't want to set an option on every invocation.

@tlimoncelli
Copy link
Copy Markdown
Contributor

Either a property of the provider (kind of like how BIND has default_ns defined (https://stackexchange.github.io/dnscontrol/providers/bind) or something listed in creds.json. That way it doesn't have to be included in the DSL over and over.

@captncraig
Copy link
Copy Markdown
Contributor

This is a great idea. I still think the best idea is for the namecheap driver to handle this. We will be adding the ability to run all domains in parallel sometime soon, and this being in main makes that a bit trickier.

I'd recommend something in the namecheap package to throttle itself. Maybe a simple time.Ticker would do:

//package level variable:
var throttle = time.NewTicker(time.Second) // or whatever delay between requests seems reasonable

//wherever we make a request:
<- throttle
http.Get(...

@tlimoncelli
Copy link
Copy Markdown
Contributor

I'm merging this. It is sufficient for the intended purpose. (i.e. a full rate-limiting solution would be grand, but it isn't going to happen any time soon. It is probably easier to work with vendors to remove/increase rate limits than to code the perfect solution)

@tlimoncelli tlimoncelli merged commit 8cff312 into StackExchange:master Aug 5, 2017
rblenkinsopp pushed a commit to rblenkinsopp/dnscontrol that referenced this pull request Aug 21, 2020
* New flag: -delay=x where x is ms to delay between domains. This feature will probably be removed some day when a better rate-limiting solution can be engineered.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants