dns-controller: allow configuring DNS update interval

[diversario]

Given enough DNS records, and with update batching, Route53 will throttle API calls. We set this in our forked DNS controller to 30 seconds to avoid throttling.

[idealhack]

/ok-to-test

[chrisz100]

I am wondering if this is the way we should go about this problem... @justinsb @chrislovecnm any thoughts?

[diversario]

This is definitely a workaround and if there's a better solution I'm all for that.

AWS R53 docs say Five requests per second per AWS account. but my understanding is that this isn't exactly how it happens – that they use a token bucket approach, and in certain configurations DNS updates become effectively not possible with the current DNS controller config. In our situation, we have several clusters (envs) in the same AWS account. All of them have a large number of records. We had to lower our batch size to keep updates under 32000 chars, and that, in turn, caused the number of requests go above the throttling limit. Lower update frequency helps us stay under the limit.

Perhaps a better approach is to space requests a bit but I think that would require modification in the R53 AWS provider.

[geojaz]

I think this makes sense- it's fine by me.

But why don't we let folks configure the interval AND set it to 30 seconds by default? Is 30 seconds an unreasonably long interval for "normal" circumstances? Rate limiting is a constant problem on AWS, I'd rather be a little bit restrained than try to hammer the API and unexpectedly break things.

[diversario]

Well, I personally have nothing against 30s by default, but I'm saying this without understanding why it is 5s currently. If 5s was chosen arbitrarily then I see no problem with increasing the delay.

[justinsb]

30 seconds would mean at least a 25 second slow-down on cluster-bring-up (vs 5 seconds), and I bet there's more than one (etcd & apiserver records).

I'd rather keep this defaulting to ~ 5 seconds; I certainly agree having a better strategy would be preferable.

I think the challenge is that the rate limit isn't easily known, so we instead back-off when we see it. If you have logs you could share that might be helpful (e.g. on slack). I know that we could be smarter about batch-size to try to pack each batch full (i.e. estimate / compute the record size).

AFAICT, you haven't exposed the flag in kops itself? I guess that means you are manually setting it; and FYI I think that means that kops will clear the flag it every time we update dns-controller. But that can be a separate PR!

/approve
/lgtm

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: diversario, justinsb

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [justinsb]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment