server does not allow access to the requested resources

[huang195]

After installing k8s 1.6 via kubeadm, I tried to install coredns to replace kube-dns, and the coredns pod/svc does get successfully deployed. However, inspecting the log of coredns shows the following errors that repeats every few seconds:

E0423 18:35:23.759412       1 reflector.go:214] github.com/coredns/coredns/vendor/k8s.io/client-go/1.5/tools/cache/reflector.go:109: Failed to list *api.Namespace: the server does not allow access to the requested resource (get namespaces)
E0423 18:35:23.832581       1 reflector.go:214] github.com/coredns/coredns/vendor/k8s.io/client-go/1.5/tools/cache/reflector.go:109: Failed to list *api.Service: the server does not allow access to the requested resource (get services)
E0423 18:35:23.833121       1 reflector.go:214] github.com/coredns/coredns/vendor/k8s.io/client-go/1.5/tools/cache/reflector.go:109: Failed to list *api.Endpoints: the server does not allow access to the requested resource (get endpoints)

I do have a default service account in the namespace kube-system:

root@radiant1:~# kubectl get serviceaccounts -n kube-system | grep default
default                      1         1d

AFAIK, this service account is what coredns uses:

  serviceAccount: default
  serviceAccountName: default

Any thoughts on this is not working?

[johnbelamaric]

@chrisohaver had this issue and has a solution. It's the RBAC in 1.6, it needs more permissions to access other namespaces. @chrisohaver?

[chrisohaver]

RBAC is used by default in k8s 1.6. The kube-system default account no longer has access by default to the API. Also, coredns requires access to a couple more API objects than kube-dns does. There are a few ways to grant access. I think the cleanest solution is to create a new ServiceAccount, ClusterRole, and ClusterRoleBinding for coredns, and then configure the pods to use the new service account.

Create ServiceAccount, ClusterRole, and ClusterRoleBinding

apiVersion: v1
kind: ServiceAccount
metadata:
  name: coredns
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:coredns
rules:
- apiGroups:
  - ""
  resources:
  - endpoints
  - services
  - pods
  - namespaces
  verbs:
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: system:coredns
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:coredns
subjects:
- kind: ServiceAccount
  name: coredns
  namespace: kube-system

Configure pods to use the coredns service account
You'll also need to point the pods to this service account. You can do this by adding serviceAccountName: coredns to the template: spec: section of the Deployment yaml.

[huang195]

@johnbelamaric @chrisohaver thanks for pointing me to a solution. This doesn't look like a problem with coredns deployment but rather an issue in k8s. I wished I saw the solution earlier as the solution I hacked up was not as clean. I ended up reusing the kube-dns service account by adding namespaces into its access list and using that instead of the default service account in the kube-system namespace. This has also worked.

[johnbelamaric]

@chrisohaver can you add those templates here? Or, better make a 1.6 and later version of the deployment manifest that includes these?