dnsmadeeasy: authorizing ~50+ domains causes a fatal rate limiting error

[alexzorin]

Originally reported on the forums.

My operating system is (include version):

Ubuntu Disco

I installed Certbot with (certbot-auto, OS package manager, pip, etc):

certbot-auto

I ran this command and it produced this output:

I ran (50 domains):

certbot-auto certonly --dns-dnsmadeeasy --dns-dnsmadeeasy-credentials /etc/letsencrypt/dnsmadeeasy-creds.ini -d "certbot-dnsmadeeasy-test.ga" -d domain-1.certbot-dnsmadeeasy-test.ga -d domain-2.certbot-dnsmadeeasy-test.ga -d domain-3.certbot-dnsmadeeasy-test.ga -d domain-4.certbot-dnsmadeeasy-test.ga -d domain-5.certbot-dnsmadeeasy-test.ga -d domain-6.certbot-dnsmadeeasy-test.ga -d domain-7.certbot-dnsmadeeasy-test.ga -d domain-8.certbot-dnsmadeeasy-test.ga -d domain-9.certbot-dnsmadeeasy-test.ga -d domain-10.certbot-dnsmadeeasy-test.ga -d domain-11.certbot-dnsmadeeasy-test.ga -d domain-12.certbot-dnsmadeeasy-test.ga -d domain-13.certbot-dnsmadeeasy-test.ga -d domain-14.certbot-dnsmadeeasy-test.ga -d domain-15.certbot-dnsmadeeasy-test.ga -d domain-16.certbot-dnsmadeeasy-test.ga -d domain-17.certbot-dnsmadeeasy-test.ga -d domain-18.certbot-dnsmadeeasy-test.ga -d domain-19.certbot-dnsmadeeasy-test.ga -d domain-20.certbot-dnsmadeeasy-test.ga -d domain-21.certbot-dnsmadeeasy-test.ga -d domain-22.certbot-dnsmadeeasy-test.ga -d domain-23.certbot-dnsmadeeasy-test.ga -d domain-24.certbot-dnsmadeeasy-test.ga -d domain-25.certbot-dnsmadeeasy-test.ga -d domain-26.certbot-dnsmadeeasy-test.ga -d domain-27.certbot-dnsmadeeasy-test.ga -d domain-28.certbot-dnsmadeeasy-test.ga -d domain-29.certbot-dnsmadeeasy-test.ga -d domain-30.certbot-dnsmadeeasy-test.ga -d domain-31.certbot-dnsmadeeasy-test.ga -d domain-32.certbot-dnsmadeeasy-test.ga -d domain-33.certbot-dnsmadeeasy-test.ga -d domain-34.certbot-dnsmadeeasy-test.ga -d domain-35.certbot-dnsmadeeasy-test.ga -d domain-36.certbot-dnsmadeeasy-test.ga -d domain-37.certbot-dnsmadeeasy-test.ga -d domain-38.certbot-dnsmadeeasy-test.ga -d domain-39.certbot-dnsmadeeasy-test.ga -d domain-40.certbot-dnsmadeeasy-test.ga -d domain-41.certbot-dnsmadeeasy-test.ga -d domain-42.certbot-dnsmadeeasy-test.ga -d domain-43.certbot-dnsmadeeasy-test.ga -d domain-44.certbot-dnsmadeeasy-test.ga -d domain-45.certbot-dnsmadeeasy-test.ga -d domain-46.certbot-dnsmadeeasy-test.ga -d domain-47.certbot-dnsmadeeasy-test.ga -d domain-48.certbot-dnsmadeeasy-test.ga -d domain-49.certbot-dnsmadeeasy-test.ga -d domain-50.certbot-dnsmadeeasy-test.ga --dry-run

It produced:

Performing the following challenges:
dns-01 challenge for certbot-dnsmadeeasy-test.ga
dns-01 challenge for domain-1.certbot-dnsmadeeasy-test.ga
dns-01 challenge for domain-10.certbot-dnsmadeeasy-test.ga
dns-01 challenge for domain-11.certbot-dnsmadeeasy-test.ga
dns-01 challenge for domain-12.certbot-dnsmadeeasy-test.ga
dns-01 challenge for domain-13.certbot-dnsmadeeasy-test.ga
dns-01 challenge for domain-14.certbot-dnsmadeeasy-test.ga
dns-01 challenge for domain-15.certbot-dnsmadeeasy-test.ga
dns-01 challenge for domain-16.certbot-dnsmadeeasy-test.ga
dns-01 challenge for domain-17.certbot-dnsmadeeasy-test.ga
dns-01 challenge for domain-18.certbot-dnsmadeeasy-test.ga
dns-01 challenge for domain-19.certbot-dnsmadeeasy-test.ga
dns-01 challenge for domain-2.certbot-dnsmadeeasy-test.ga
dns-01 challenge for domain-20.certbot-dnsmadeeasy-test.ga
dns-01 challenge for domain-21.certbot-dnsmadeeasy-test.ga
dns-01 challenge for domain-22.certbot-dnsmadeeasy-test.ga
dns-01 challenge for domain-23.certbot-dnsmadeeasy-test.ga
dns-01 challenge for domain-24.certbot-dnsmadeeasy-test.ga
dns-01 challenge for domain-25.certbot-dnsmadeeasy-test.ga
dns-01 challenge for domain-26.certbot-dnsmadeeasy-test.ga
dns-01 challenge for domain-27.certbot-dnsmadeeasy-test.ga
dns-01 challenge for domain-28.certbot-dnsmadeeasy-test.ga
dns-01 challenge for domain-29.certbot-dnsmadeeasy-test.ga
dns-01 challenge for domain-3.certbot-dnsmadeeasy-test.ga
dns-01 challenge for domain-30.certbot-dnsmadeeasy-test.ga
dns-01 challenge for domain-31.certbot-dnsmadeeasy-test.ga
dns-01 challenge for domain-32.certbot-dnsmadeeasy-test.ga
dns-01 challenge for domain-33.certbot-dnsmadeeasy-test.ga
dns-01 challenge for domain-34.certbot-dnsmadeeasy-test.ga
dns-01 challenge for domain-35.certbot-dnsmadeeasy-test.ga
dns-01 challenge for domain-36.certbot-dnsmadeeasy-test.ga
dns-01 challenge for domain-37.certbot-dnsmadeeasy-test.ga
dns-01 challenge for domain-38.certbot-dnsmadeeasy-test.ga
dns-01 challenge for domain-39.certbot-dnsmadeeasy-test.ga
dns-01 challenge for domain-4.certbot-dnsmadeeasy-test.ga
dns-01 challenge for domain-40.certbot-dnsmadeeasy-test.ga
dns-01 challenge for domain-41.certbot-dnsmadeeasy-test.ga
dns-01 challenge for domain-42.certbot-dnsmadeeasy-test.ga
dns-01 challenge for domain-43.certbot-dnsmadeeasy-test.ga
dns-01 challenge for domain-44.certbot-dnsmadeeasy-test.ga
dns-01 challenge for domain-45.certbot-dnsmadeeasy-test.ga
dns-01 challenge for domain-46.certbot-dnsmadeeasy-test.ga
dns-01 challenge for domain-47.certbot-dnsmadeeasy-test.ga
dns-01 challenge for domain-48.certbot-dnsmadeeasy-test.ga
dns-01 challenge for domain-49.certbot-dnsmadeeasy-test.ga
dns-01 challenge for domain-5.certbot-dnsmadeeasy-test.ga
dns-01 challenge for domain-50.certbot-dnsmadeeasy-test.ga
dns-01 challenge for domain-6.certbot-dnsmadeeasy-test.ga
dns-01 challenge for domain-7.certbot-dnsmadeeasy-test.ga
dns-01 challenge for domain-8.certbot-dnsmadeeasy-test.ga
dns-01 challenge for domain-9.certbot-dnsmadeeasy-test.ga
Cleaning up challenges
Error determining zone identifier: 400 Client Error: Bad Request for url: https://api.sandbox.dnsmadeeasy.com/V2.0/dns/managed/name?domainname=certbot-dnsmadeeasy-test.ga.

Certbot's behavior differed from what I expected because:

Provider's published rate limits are as follows:

To prevent unwanted flooding of the API system, there is a maximum number of requests that can be sent in a given time period. This limit is 150 requests per 5 minute scrolling window

The pattern of requests generated by Certbot/Lexicon (3 requests per domain) means that having a reasonably high number of domains (50) will guarantee a fatal error during runtime, either during auth or cleanup. Math in my head tells me that 25 domains should be enough to trigger the problem.

The rate limit response is an HTTP 400 with a response body of:

{"error": ["Rate limit exceeded"]}

but the body isn't logged by Certbot or Lexicon, and is just reported as a generic 400.

Here is a Certbot log showing the issue (if available):

letsencrypt.log

[djmetzle]

I would suggest either warning in the dnsmadeeasy plugin on >50 domains, or directly emitting 429 rate-limit message from lexicon. Or both.

[alexzorin]

A simple fix could be just to sleep when we see that the xdnsme-requestsRemaining response header indicates that the next request is going to fail, and then occasionally retry. It's kind of a bummer that there is no differentiation between 400 and 400 (rate limited).

Certbot isn't exposed to that level of detail, so that'd need to be part of Lexicon's behavior. WDYT @AnalogJ?

[adferrand]

Another Lexicon maintainer here.

Yes, I think it is doable at Lexicon level. However we can go a little further. Indeed a sleep on its own does not guarantee that you moved out of the limit window, and so that the next request after the sleep will not fail. Except if you put a 5 min sleep, but it is a big waste since new requests are certainly possible after some seconds only.

So the API will return a 400, but with an informative body. This is sufficient to recognize it among other potential 400 responses. How about defining a short sleep (like 5 seconds), and a retry strategy with a max attempts, triggered in case of 400 with body {"error": ["Rate limit exceeded"]}?

@alexzorin I would suggest you to open a new issue on Lexicon repository, linking to this one. I do not have credentials for dnsmadeeasy, but if someone can provide me temporary ones, I would be glad to implement this retry logic.

[adferrand]

I would note however that this is a workaround that is not solving the root cause in my opinion: 150 requests on a 5 min window is way too low for an API, typically designed for automation. I would also suggest to contact dnsmadeeasy maintainers to largely relax this limit.

[alexzorin]

@adferrand I have some DME sandbox credentials you can use which has certbot-dnsmadeeasy-test.ga's DNS hosted with it - how can I send them to you?

[adferrand]

I propose to move to the dedicated Lexicon issue for further discussions.

For Certbot's concern, this issue can be closed once AnalogJ/lexicon#437 is fixed.

[bmw]

I'm closing this in favor of #7631.

This issue is resolved by using a new version of our dependency dns-lexicon. #7631 is tracking updating the version of the library we have pinned so it will be updated everywhere we have control over our dependency versions like in our Docker images.

If you're having this issue using packages provided by your OS, I'd ask your distro maintainers to update the packaged version of dns-lexicon.