CoreDNS Autoscaler

[davidnuzik]

CoreDNS Autoscaler work for Rancher 2.6.x

[cjellick]

RKE1 did this. @superseb can give more context. Or perhaps @Oats87 can.

If feasible, we should also do in k3s.

[manuelbuil]

Need some clarification. The code is already present in the coredns chart that rke2 is using but the variable to enable this feature was set to false. Should I enable it by default?

[manuelbuil]

Apart from that, I guess the tasks for this issue are:

• Use an internal rancher image for the autoscaler (e.g. what rke is using) instead of the upstream image
• Update the coredns hardened image version (1.6.9) so that it is aligned with the rke autoscaler image (1.8.3)
• Document what is the dns autoscaler and how to enable/disable it
• Add new images into rke2 airgap
• Update coredns helm chart
• Enable it by default? Yes. https://rancher.slack.com/archives/CRF4B1J8H/p1626792513234700

[manuelbuil]

Tested and works. To test:

• Verify there is a new deployment : rke2-coredns-rke2-coredns-autoscaler on namespace kube-system
• Verify there is a new configMap: rke2-coredns-rke2-coredns-autoscaler on namespace kube-system
• Check the logs of the autoscaler and verfy there are no errors. Working logs:

I0720 09:38:52.055732       1 autoscaler.go:49] Scaling Namespace: kube-system, Target: deployment/rke2-coredns-rke2-coredns
I0720 09:38:53.056094       1 plugin.go:50] Set control mode to linear
I0720 09:38:53.056343       1 linear_controller.go:60] ConfigMap version change (old:  new: 1967) - rebuilding params
I0720 09:38:53.056532       1 linear_controller.go:61] Params from apiserver: 
{
  "coresPerReplica": 256,
  "nodesPerReplica": 16,
  "preventSinglePointFailure": true
}
I0720 09:38:53.056771       1 linear_controller.go:80] Defaulting min replicas count to 1 for linear controller

[galal-hussein]

Verified against master commit ID: 09bb5c2

Steps to verify:

• Install rke2 with commit ID:

curl https://get.rke2.io | INSTALL_RKE2_COMMIT=09bb5c27f29cf3dd831653543d274c948a225385 sh -

• verified that autoscaler deployement exists in kube-system namespace:

root@ip-172-31-15-16:~# kubectl get deployments -n kube-system
NAME                                   READY   UP-TO-DATE   AVAILABLE   AGE
rke2-coredns-rke2-coredns              1/1     1            1           8m39s
rke2-coredns-rke2-coredns-autoscaler   1/1     1            1           8m39s
rke2-metrics-server                    1/1     1            1           7m59s

• verified that configmap created for autoscaler in kube-system namespace:

kubectl get configmap -n kube-system rke2-coredns-rke2-coredns-autoscaler -o yaml
apiVersion: v1
data:
  linear: |-
    {
      "coresPerReplica": 256,
      "nodesPerReplica": 16,
      "preventSinglePointFailure": true,
      "min": 0,
      "max": 0,
      "includeUnschedulableNodes": false
    }
kind: ConfigMap
metadata:
  annotations:
    meta.helm.sh/release-name: rke2-coredns
    meta.helm.sh/release-namespace: kube-system
  creationTimestamp: "2021-09-22T20:21:03Z"
  labels:
    app.kubernetes.io/instance: rke2-coredns
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: rke2-coredns-autoscaler
    helm.sh/chart: rke2-coredns-1.16.201-build2021072308
    k8s-app: kube-dns-autoscaler
    kubernetes.io/cluster-service: "true"
    kubernetes.io/name: CoreDNS
  name: rke2-coredns-rke2-coredns-autoscaler
  namespace: kube-system
  resourceVersion: "586"
  uid: 1e270506-9937-41c0-9732-83d7f909526f

• verified that there are no error logs in the coredns pod:

kubectl logs -n kube-system rke2-coredns-rke2-coredns-autoscaler-7c58bd5b6c-qk2rl
I0922 20:21:42.401801       1 autoscaler.go:49] Scaling Namespace: kube-system, Target: deployment/rke2-coredns-rke2-coredns
I0922 20:21:42.752449       1 plugin.go:50] Set control mode to linear
I0922 20:21:42.752473       1 linear_controller.go:60] ConfigMap version change (old:  new: 586) - rebuilding params
I0922 20:21:42.752479       1 linear_controller.go:61] Params from apiserver: 
{
  "coresPerReplica": 256,
  "nodesPerReplica": 16,
  "preventSinglePointFailure": true,
  "min": 0,
  "max": 0,
  "includeUnschedulableNodes": false
}
I0922 20:21:42.752547       1 linear_controller.go:80] Defaulting min replicas count to 1 for linear controller

[galal-hussein]

Verified against rke2 v1.21.5-rc2+rke2r1

Steps to verify:

• Install rke2 server version v1.21.5-rc2+rke2r1
• Verified that deployment exists for coredns autoscaler:

root@ip-172-31-15-16:~# kubectl get deployments -n kube-system
NAME                                   READY   UP-TO-DATE   AVAILABLE   AGE
rke2-coredns-rke2-coredns              1/1     1            1           15m
rke2-coredns-rke2-coredns-autoscaler   1/1     1            1           15m
rke2-metrics-server                    1/1     1            1           14m

• Verified that Configmap exists for coredns autoscaler:

# kubectl get configmap -n kube-system rke2-coredns-rke2-coredns-autoscaler -o yaml
apiVersion: v1
data:
  linear: |-
    {
      "coresPerReplica": 256,
      "nodesPerReplica": 16,
      "preventSinglePointFailure": true,
      "min": 0,
      "max": 0,
      "includeUnschedulableNodes": false
    }
kind: ConfigMap
metadata:
  annotations:
    meta.helm.sh/release-name: rke2-coredns
    meta.helm.sh/release-namespace: kube-system
  creationTimestamp: "2021-09-22T20:38:33Z"
  labels:
    app.kubernetes.io/instance: rke2-coredns
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: rke2-coredns-autoscaler
    helm.sh/chart: rke2-coredns-1.16.201-build2021072308
    k8s-app: kube-dns-autoscaler
    kubernetes.io/cluster-service: "true"
    kubernetes.io/name: CoreDNS
  name: rke2-coredns-rke2-coredns-autoscaler
  namespace: kube-system
  resourceVersion: "580"
  uid: a4cc6245-d6cf-4f8e-b773-dd40f0016764

• Verified that there are no error logs in the coredns pod:

# kubectl logs rke2-coredns-rke2-coredns-autoscaler-7c58bd5b6c-vndcd -n kube-system
I0922 20:39:07.103041       1 autoscaler.go:49] Scaling Namespace: kube-system, Target: deployment/rke2-coredns-rke2-coredns
I0922 20:39:07.596034       1 plugin.go:50] Set control mode to linear
I0922 20:39:07.596173       1 linear_controller.go:60] ConfigMap version change (old:  new: 580) - rebuilding params
I0922 20:39:07.596205       1 linear_controller.go:61] Params from apiserver: 
{
  "coresPerReplica": 256,
  "nodesPerReplica": 16,
  "preventSinglePointFailure": true,
  "min": 0,
  "max": 0,
  "includeUnschedulableNodes": false
}
I0922 20:39:07.596363       1 linear_controller.go:80] Defaulting min replicas count to 1 for linear controller