dns_route53: add change-max-poll config argument.

[ezekiel]

This allows adjustment of the formerly hardcoded 120-rounds value.

To timeout more quickly, adjust down:
--dns-route53-change-max-poll 100

To timeout more slowly, adjust up:
--dns-route53-change-max-poll 150

[jsha]

Before we add a new command line flag, I'd like to try and find the root cause of why INSYNC status sometimes takes a long time. If slow syncing is caused by throttling at Route 53, instead of allowing higher poll times we should present the user with a message informing them that they may be encountering throttling and should adjust their issuance intervals.

[ezekiel]

applicable limits documented for route53:

All requests

Five requests per second per AWS account. If you submit more than five requests per second, Amazon Route 53 returns an HTTP 400 error (Bad request). The response header also includes a Code element with a value of Throttling and a Message element with a value of Rate exceeded.

ChangeResourceRecordSets requests

If Route 53 can't process a request before the next request arrives, it will reject subsequent requests for the same hosted zone and return an HTTP 400 error (Bad request). The response header also includes a Code element with a value of PriorRequestNotComplete and a Message element with a value of The request was rejected because Route 53 was still processing a prior request.

In both cases, amazon says it will return HTTP 400. I'll double-check if any of my logs show such an error.

[ezekiel]

Reviewing the response headers logged by certbot and neither Throttling nor PriorRequestNotComplete were ever the cause of timeouts for my systems. The change requests remained in PENDING status for as long as 25 minutes.

I think there are just some semi-rare occasions where synchronization is taking much longer than expected within Amazon infrastructure.

[arcivanov]

related to #6125

[ohemorange]

Hey @jsha, could you take a look at this?

[jsha]

We wound up working around this by reducing the number of times we update DNS. Closing since fewer options > more options. We might want to consider adding a message when polling for a change times out saying something like "If you've updated a lot of times recently, you might be getting rate limited by Amazon." But I'd want to see some more reports from other users first.

[jsha]

A little more detail here: @ezekiel reminds me that we never actually found evidence that we were being rate limited, so that might not be the root cause.

According to Amazon docs,

Amazon Route 53 is designed to propagate updates you make to your DNS records to its world-wide network of authoritative DNS servers within 60 seconds under normal conditions.

It's not clear how long they are outside "normal conditions," but the current setting will wait up to 10 minutes, which seems sufficiently generous. If anyone else has this issue, however, please let us know and provide details of your setup. We'd love to know why some updates take a long time to sync.