"The Service "kube-dns" is invalid" on fresh install

[AdamIsrael]

inspection-report-20190806_185828.tar.gz

During a fresh installation on Bionic with microk8s v1.15.2, I received the following when enabling the dns plugin:

$ microk8s.enable dashboard storage dns
Applying manifest
secret/kubernetes-dashboard-certs created
serviceaccount/kubernetes-dashboard created
role.rbac.authorization.k8s.io/kubernetes-dashboard-minimal created
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard-minimal created
deployment.apps/kubernetes-dashboard created
service/kubernetes-dashboard created
service/monitoring-grafana created
service/monitoring-influxdb created
service/heapster created
deployment.extensions/monitoring-influxdb-grafana-v4 created
serviceaccount/heapster created
clusterrolebinding.rbac.authorization.k8s.io/heapster created
configmap/heapster-config created
configmap/eventer-config created
deployment.extensions/heapster-v1.5.2 created

If RBAC is not enabled access the dashboard using the default token retrieved with:

token=$(microk8s.kubectl -n kube-system get secret | grep default-token | cut -d " " -f1)
microk8s.kubectl -n kube-system describe secret $token

In an RBAC enabled setup (microk8s.enable RBAC) you need to create a user with restricted
permissions as shown in https://github.com/kubernetes/dashboard/wiki/Creating-sample-user

Enabling default storage class
deployment.extensions/hostpath-provisioner created
storageclass.storage.k8s.io/microk8s-hostpath created
serviceaccount/microk8s-hostpath created
clusterrole.rbac.authorization.k8s.io/microk8s-hostpath created
clusterrolebinding.rbac.authorization.k8s.io/microk8s-hostpath created
Storage will be available soon
Enabling DNS
Applying manifest
serviceaccount/coredns created
configmap/coredns created
deployment.apps/coredns created
clusterrole.rbac.authorization.k8s.io/coredns created
clusterrolebinding.rbac.authorization.k8s.io/coredns created
The Service "kube-dns" is invalid: spec.clusterIP: Invalid value: "10.152.183.10": provided IP is already allocated

That IP address is already assigned to heapster:

$ microk8s.kubectl get services -n kube-system
NAME                   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
heapster               ClusterIP   10.152.183.10    <none>        80/TCP              20m
kubernetes-dashboard   ClusterIP   10.152.183.151   <none>        443/TCP             20m
monitoring-grafana     ClusterIP   10.152.183.72    <none>        80/TCP              20m
monitoring-influxdb    ClusterIP   10.152.183.115   <none>        8083/TCP,8086/TCP   20m

microk8s.status reports that DNS is enabled, but I can't edit its configmap:

$ microk8s.kubectl -n kube-system edit configmap/kube-dns
Error from server (NotFound): configmaps "kube-dns" not found

Disabling DNS fails:

$ microk8s.disable dns
Disabling DNS
Reconfiguring kubelet
Removing DNS manifest
serviceaccount "coredns" deleted
configmap "coredns" deleted
deployment.apps "coredns" deleted
clusterrole.rbac.authorization.k8s.io "coredns" deleted
clusterrolebinding.rbac.authorization.k8s.io "coredns" deleted
Error from server (NotFound): error when deleting "/home/ubuntu/snap/microk8s/743/tmp/temp.coredns.yaml": services "kube-dns" not found

Attempting to re-enable DNS results in the same error:

$ microk8s.enable dns
Enabling DNS
Applying manifest
serviceaccount/coredns created
configmap/coredns created
deployment.apps/coredns created
clusterrole.rbac.authorization.k8s.io/coredns created
clusterrolebinding.rbac.authorization.k8s.io/coredns created
The Service "kube-dns" is invalid: spec.clusterIP: Invalid value: "10.152.183.10": provided IP is already allocated

[ktsakalozos]

Thank you for reporting this @AdamIsrael. I am sure you see that the dns addon, being an important service for the cluster expects to get the 10.152.183.10 cluster IP. Unfortunately heapster got that IP first. For sure this is a race condition we need to address somehow.

[balchua]

We somehow hardcode the clusterip in the kube-dns service manifest. Just wondering is it necessary to do so?

[ktsakalozos]

We would need to remove the hardcoded cluster ip in https://github.com/ubuntu/microk8s/blob/master/microk8s-resources/actions/coredns.yaml#L151 and do a query to the API server for the appointed IP in the enable script https://github.com/ubuntu/microk8s/blob/master/microk8s-resources/actions/enable.dns.sh#L24

[balchua]

Thanks. If the cluster-dns is set on the kubelet on the get go even without enabling dns, would it still give the same ip to newly created services? Sorry for the lame question. 😁

[ktsakalozos]

The cluster IPs are not appointed by kubelet so I assume you cannot reserve an IP like this (pre-configuring kubelet). Also if we run with a kubelet pre-configured with a cluster-dns it would inject that dns ip into all created pods causing problems when the pods try to resolve anything and the dns is nor enabled.

[balchua]

@ktsakalozos oh i get it now. Thanks for the insight. What about enabling dns by default upon installing microk8s. I think most apps needs the dns anyway.

[davigar15]

Any updates on this?

[ktsakalozos]

@davigar15 the fix described in #585 (comment) has low priority as the workaround is to do the microk8s.enable dns first. If there is however a PR we can merge it.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.