promote coredns 1.8.4 in kube 1.22

[pacoxu]

As far as I know, before that, we should update https://github.com/coredns/corefile-migration as well for kubeadm upgrading or so.

• Promote CoreDNS v1.8.3 Promote CoreDNS v1.8.3 k8s.io#1725
• WIP: Promote CoreDNS v1.8.4 (and dns cache 1.18.0) promote coredns 1.8.4 and dns cache 1.18.0 k8s.io#2118
• • node-local-dns 1.18.0: https://github.com/kubernetes/dns/releases/tag/1.18.0 : include debian to 1.6.0; dnsmasq-2.78 upgrade to 2.85
• 1.8.3 migration Update up through 1.8.3 coredns/corefile-migration#59 Done by @chrisohaver
• WIP: migration support in add upgrade support for 1.8.4 coredns/corefile-migration#60 1.8.4
• kubeadm support for 1.8.3 ([kubeadm]: Bump CoreDNS version to 1.8.0 #96429 is an example for 1.8.0 support )
• kubeadm 1.8.4 kubeadm: update coredns to 1.8.4 #102466
  next step：
• Open a new issue for next stable coredns version(1.9.2 maybe) tracking. upgrade coredns to 1.8.6 to fix a regression bug #105189 for 1.8.5

As 1.21 code freeze is next week, I'm not sure whether we should support it in this release or next.

I also checked the node local dns part.

• follow up of the debian CVE update is in update debian image tag 1.4.0 dns#441
• 1.17.0 has something valuable for IPv6 support. update node local dns to 1.17.0 for IPv6 support/hosts/trace plugins #99749 (1.21) (the last promotion to k8s.io is 1.17.0 Promote 1.17 DNS images k8s.io#1640)
• update node-local-dns readme update node local dns readme for ipv6 and 1.18 GA #101417

https://github.com/ldir-EDB0/dnsmasq/blob/master/CHANGELOG#L111-L114

Cleanup of kube-dns:

• remove kube-dns support in kubeadm
• My mistake, should not be removed/skipped: skip kube-dns auto-scaling in e2e e2e-kops: skip kube-dns autoscaling test after 1.20 #100882

BTW, is there a plan to remove https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/dns/kube-dns as deprecated? (kube-dns rbac related issue #60897)

What would you like to be added:

Currently, the coredns using in Kubernetes is 1.7.0.
The newer version includes 1.7.1\1.8.0\1.8.3\1.8.4.
The kubeadm supports 1.8.0 so I did a cleanup in #

Why is this needed:

I go through the release notes in coredns/coredns. More info can be found in https://github.com/coredns/coredns/blob/master/notes/coredns-1.8.3.md

• Corrected detection of K8s minor version Corrected detection of K8s minor version coredns/coredns#4430 This is something good to have.
• Node Local DNS debian cve update is needed.

/cc @prameshj @rajansandeep

Not in 1.22 as it is not merged yet:

• non root for coredns is WIP Use distroless base and also run coredns as nonroot. coredns/coredns#4528

[prameshj]

Thanks for taking this up! I have reviewed some of the DNS PRs.

BTW, is there a plan to remove https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/dns/kube-dns as deprecated?
Not yet.

[chrisohaver]

There is a bug in CoreDNS 1.8.3 (and 1.8.1) that causes CoreDNS to fail to start while the k8s API is down. Eventually once the k8s API is up, CoreDNS will start. I don't know if this is severe enough of an issue to hold back until it is fixed. It would at least have the potential to cause a notable delay in CoreDNS starting, due to crashloop backoff. I don't know if it could cause more serious bootstrap type issues (wherein some process relies on CoreDNS forwarding queries out of cluster before the API can start).

[neolit123]

@pacoxu

As 1.21 code freeze is next week, I'm not sure whether we should support it in this release or next.

i think it's too late for bumping coredns for 1.21 at this point.
i'm basing this statement on contributor / reviewer bandwidth from the past.

BTW, is there a plan to remove https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/dns/kube-dns as deprecated?

the cluster/addons folder used to be owned by SIG Cluster Lifecycle, but now this falls under the GCP cloud provider as they consume the addon installer for testing. from my POV the public use of addons/* is depracated.

@chrisohaver

There is a bug in CoreDNS 1.8.3 (and 1.8.1) that causes CoreDNS to fail to start while the k8s API is down. Eventually once the k8s API is up, CoreDNS will start

thanks for the info, this contributes to my vote to delay the update.
(even if it doesn't look like a severe problem)

[pacoxu]

• I will start to test coredns 1.8.3 recently. It may targets for 1.22.
• I prefer to upgrade node-local-dns for IPv6 support in 1.21. update node local dns to 1.17.0 for IPv6 support/hosts/trace plugins #99749 is merged

[neolit123]

/sig network cluster-lifecycle
/triage accepted

[pacoxu]

/sig network
/area dns

[chrischdi]

Hi,

I hope I don't disturb on this issue but maybe I have relevant information.

We already planned to make use of coredns 1.8.3. However we found an issue regarding headless services.

Beginning with coredns 1.8.3 (I did not find any images for 1.8.1 or 1.8.2 to test), coredns does still resolve names of headless services, while the pods itself is not yet marked as ready.

Am I missing something here or does this look like a bug or breaking change?

I have a gist here to reproduce it using kind, which also shows the different behaviour between 1.8.0 and 1.8.3.

Edit: I forgot to mention that the description of svc.spec.publishNotReadyAddresses explicitly says that this behaviour could be toggled.

publishNotReadyAddresses indicates that any agent which deals with
endpoints for this Service should disregard any indications of
ready/not-ready. The primary use case for setting this field is for a
StatefulSet's Headless Service to propagate SRV DNS records for its Pods
for the purpose of peer discovery.

[pacoxu]

@chrischdi
I will confirm/test it.
Not in upgrade coredns yet. We may have to move to a stable version laer if so.

[chrisohaver]

Beginning with coredns 1.8.3 (I did not find any images for 1.8.1 or 1.8.2 to test), coredns does still resolve names of headless services, while the pods itself is not yet marked as ready.

Am I missing something here or does this look like a bug or breaking change?

Thanks for identifying it!, Yes, it's a bug, introduced with the move from the Endpoints API to EndpointSlices API.

I opened a PR to fix it. coredns/coredns#4580

[chrischdi]

@pacoxu : we are also running the e2e conformance tests. Shouldn't there be a test for this (something like not ready headless service should not resolve when pod is not ready)? I'd be happy to contribute :-)

[aojea]

/cc

[BenTheElder]

cluster/ is still necessary for CI for the immediate future, we should continue to minimally maintain there for the purposes of kubernetes CI signal. It's not really relevant to GCP, we just run it in GCP because:

• the jobs exist and work
• we have the credits necessary to do e.g. scale testing
• there is no comparable CI. Kind alone is insufficient, and we would want different CI to compare against anyhow

I disagree that this is SIG cloud provider. It's SIG everyone that wants e2e coverage beyond kind.

[BenTheElder]

it's also important to note that hack/local-up-cluster.sh shares these addons, which is very much used for local development by contributors e.g. @dims @liggitt, we should not let the coreDNS addon lapse

kubernetes/hack/local-up-cluster.sh

Line 42 in 3a47ddc

DNS_ADDON=${DNS_ADDON:-"coredns"}

kubernetes/hack/local-up-cluster.sh

Line 949 in 3a47ddc

cp "${KUBE_ROOT}/cluster/addons/dns/${DNS_ADDON}/${DNS_ADDON}.yaml.in" dns.yaml

[pacoxu]

I made a mistake here. I thought that the dns-autoscaler only supports old kube-dns, however, it supports both kube-dns and coredns.

1. one practice is running coredns deployment with 2 or 3 replicas, plus node-local-dns daemonset(This is GAed https://kubernetes.io/docs/tasks/administer-cluster/nodelocaldns/)
2. another practice is coredns deployment, plus dns-auto-scaler. (I missed this practice. https://kubernetes.io/docs/tasks/administer-cluster/dns-horizontal-autoscaling/)
3. the old practice is Kube-dns deployment, plus dns-auto-scaler.(this is deprecated)

I am not sure which is the best practice. Or we should use coredns + node-local-dns + dns auto-scaler.

BTW, I will look into #100795 later for the dns-autoscaling case failure before.

[chrisohaver]

I am not sure which is the best practice. Or we should use coredns + node-local-dns + dns auto-scaler.

My 2 cents: I don't know about best practice, but theoretically it does makes sense to use the cluster proportional auto-scaler in conjunction with a coredns + node-local-dns deployment. However the cores/nodes per replica value should be set much higher, since the load of caching and upstream forwarding is distributed to the node local dns instances. The amount of load distributed would be dependent on the ratio of in-cluster to out-of-cluster dns name lookups, and in-cluster cache hit ratio, which could be highly variable among different clusters.

https://kubernetes.io/docs/tasks/administer-cluster/dns-horizontal-autoscaling/ doesn't provide guidance on what the value should be. It just says "Modify the fields according to your needs."

[thockin]

Did this get done?

[pacoxu]

Yes I think