Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
change default 5s ttl to 30s for coredns to be same with kube-dns/dnsmasq #76238
What type of PR is this?
What this PR does / why we need it:
According to https://coredns.io/plugins/kubernetes/, coredns uses 5s ttl for kubernetes zone by default, this isn't consistent with 30s ttl in dnsmasq based kube-dns. Although https://github.com/kubernetes/dns/blob/master/docs/specification.md doesn't specify the value for ttl, it's better to keep it same, actually this issue confused me a lot when I did benchmark against two kubernetes cluster, one uses dnsmasq, one uses coredns, I spent a lot of time to locate the root cause.
Special notes for your reviewer:
I use kube-dns only for k8s service, I'm not sure whether this change is appropriate for other kinds of k8s resources such as k8s pod.
Does this PR introduce a user-facing change?:
Hi @Dieken. Thanks for your PR.
I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with
Once the patch is verified, the new status will be reflected by the
I understand the commands that are listed here.
Longer ttl does lead to stale DNS record, but this is a well-known problem of DNS system, 5s ttl is not a good default value for DNS record.
@neolit123 I added a release note to the PR description.
I would like to just use 30s because kube-dns has used it for a long time and seems no big complaint yet. For most use cases, default 30s is good enough.
Actually k8s default coredns configuration overrides TTL for external DNS records which is very bad, dnsmasq correctly respects upstream DNS servers' TTL.
Yes. But CoreDNS has used TTL = 5 for many k8s releases, I think since 1.9. We actually had a few issues opened saying that the ttl is too long (i.e. that we cache outdated records for too long). For this reason we now allow users to set TTL to 0 so the records do not enter the cache at all, but we left the 5 as a default.
But cluster DNS is not a general DNS use case. Cluster DNS provides records for micro-services, which can change much more rapidly than general DNS applications. I don't see how short ttl has a huge performance impact on the service - i think most services always look up a name, regardless of the TTL value in the response it got last time, i.e. they don't keep a local cache.
I agree that cluster IP services don't change often. This is however only one type of record/response. a 30 TTL will also make negative responses and endpoint records cache for 30 seconds.
Sure, there is still a problem with 5... but 6X less of a problem.
What other k8s service discovery options are there? CoreDNS already integrates with the kubernetes apiserver. It keeps open a watch so it can update records quickly when they change. The 5 second TTL already partly nullifies this. A 30 second TTL would result in leaving outdated information in the cache even longer.
I guess most k8s users are still using k8s < 1.11 which uses kube-dns by default, in my company we touched k8s-1.11 several months ago :-) Those users will suddenly suffer from performance issue, unfortunately I'm one of them, I spent a lot of time to debug DNS resolution failure under high QPS and latency increase.
Of course some people would like small TTL even zero, some people would like longer TTL, there is no right or wrong, I just hope no surprise, and wonder whether 5s default TTL is good for most people.
Java has DNS cache, nodejs doesn't have, many people complained that, that's why https://www.npmjs.com/package/dnscache and https://github.com/eduardbcom/lookup-dns-cache are created, actually the most important feature of dnsmasq is DNS cache.
Somebody reported QPS increases 100 times with local DNS cache compared to no local cache (Sorry I lose the source url), in my company we find 3 times QPS increase, it could be higher if we resolve bottlenecks in other places.
Local DNS cache really matters to performance.
It's said some Java developers created homebrew Java-only RPC solution which directly integrates with k8s apiserver, this isn't mainstream usage in kubernetes community.
For my company, DNS based service discovery is good enough, we often update k8s deployments but rarely update k8s services, so the service IP is basically stable, I guess this is also true for most k8s users, this is why I suggest 30s for default TTL, we can easily realize TTL is too long and
[APPROVALNOTIFIER] This PR is APPROVED
The full list of commands accepted by this bot can be found here.
The pull request process is described here