Failed to create load balancer: User "system:kube-controller-manager" cannot create configmaps in the namespace "kube-system".

[Dirbaio]

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT

Versions

kubeadm version 1.7.5

Environment:

• Kubernetes version 1.7.5
• Cloud provider or hardware configuration: GCE (with multizone on)
• OS Ubuntu 16.04
• Kernel Linux vk-prod4-master 4.10.0-32-generic The discovery interface in the API needs to be refactored #36~16.04.1-Ubuntu SMP Wed Aug 9 09:19:02 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
• Others: Calico networking

What happened?

• Deployed k8s master like this:

cat > kubeadm.yaml <<EOF
apiVersion: kubeadm.k8s.io/v1alpha1
kind: MasterConfiguration
api:
  advertiseAddress: $MASTER_HOSTNAME
  bindPort: 443
networking:
  serviceSubnet: 10.1.0.0/16
  podSubnet: 10.0.0.0/16
kubernetesVersion: v1.7.5
cloudProvider: gce
apiServerExtraArgs:
  runtime-config: batch/v2alpha1=true
  oidc-client-id: BLAHBLAHBLAH.apps.googleusercontent.com  # beta
  oidc-issuer-url: https://accounts.google.com
  oidc-username-claim: email
controllerManagerExtraArgs:
  configure-cloud-routes: "true"
apiServerCertSANs:
- $CLUSTER_NAME.example.com
token: $CLUSTER_TOKEN
EOF

# --skip-preflight-checks because kubeadm doesn't like docker overlay2 yet
kubeadm init --skip-preflight-checks --config kubeadm.yaml

• Deployed k8s node like this:

kubeadm join \
    --skip-preflight-checks \
    --token $CLUSTER_TOKEN \
    $MASTER_HOST:443

• Everything in the cluster comes up fine and healthy.
• Create a LoadBalancer service like this:

apiVersion: v1
kind: Service
metadata:
  name: nginx-ingress
  namespace: nginx-ingress
spec:
  ports:
    - name: http
      port: 80
    - name: https
      port: 443
  selector:
    app: nginx-ingress
  type: LoadBalancer
  externalTrafficPolicy: Local
  loadBalancerIP: X.Y.Z.W

• Wait for a bit
• Load balancer fails to be created. kubectl describe svc shows:

  FirstSeen	LastSeen	Count	From			SubObjectPath	Type		Reason				Message
  ---------	--------	-----	----			-------------	--------	------				-------
  1m		15s		5	service-controller			Normal		CreatingLoadBalancer		Creating load balancer
  1m		15s		5	service-controller			Warning		CreatingLoadBalancerFailed	Error creating load balancer (will retry): Failed to create load balancer for service nginx-ingress/nginx-ingress: User "system:kube-controller-manager" cannot create configmaps in the namespace "kube-system". (post configmaps)

What you expected to happen?

Load balancer gets created successfully.

How to reproduce it (as minimally and precisely as possible)?

See above

Anything else we need to know?

This seems to be caused to missing RBAC roles/bindings. As a workaround, applying the following and recreating the service makes it work:

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: rbac-fix
  namespace: kube-system
rules:
- apiGroups:
  - ""
  resources:
  - configmaps
  verbs:
  - "*"
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: rbac-fix
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: rbac-fix
subjects:
- kind: User
  name: system:kube-controller-manager
  namespace: kube-system

[shashidharatd]

I too stumbled upon the exact same issue.
/cc @luxas @pipejakob

[shashidharatd]

/cc @kubernetes/sig-cluster-lifecycle-bugs

[shashidharatd]

Here is the log on why servicecontroller is failing to create 'LoadBalancer' type service in kubeadm based cluster

{"log":"E0909 17:54:54.712198       1 servicecontroller.go:753] Failed to process service. Retrying in 2m40s: Failed to create load balancer for service federation-system/e2e-federation-apise
rver: User \"system:kube-controller-manager\" cannot create configmaps in the namespace \"kube-system\". (post configmaps)\n","stream":"stderr","time":"2017-09-09T17:54:54.712261219Z"}
{"log":"E0909 17:57:34.716929       1 servicecontroller.go:753] Failed to process service. Retrying in 5m0s: Failed to create load balancer for service federation-system/e2e-federation-apiser
ver: User \"system:kube-controller-manager\" cannot create configmaps in the namespace \"kube-system\". (post configmaps)\n","stream":"stderr","time":"2017-09-09T17:57:34.717073975Z"}

[luxas]

We should loop in sig-auth on this as the problem is missing RBAC bindings. I guess that this might be GCE-specific, and hence not present in the general RBAC rules Can you link to the code in the gce servicecontroller that tries to modify configmaps please?

…

On 09 Sep 2017, at 21:09, shashidharatd ***@***.***> wrote: Here is the log on why servicecontroller is failing to create 'LoadBalancer' type service in kubeadm based cluster {"log":"E0909 17:54:54.712198 1 servicecontroller.go:753] Failed to process service. Retrying in 2m40s: Failed to create load balancer for service federation-system/e2e-federation-apise rver: User \"system:kube-controller-manager\" cannot create configmaps in the namespace \"kube-system\". (post configmaps)\n","stream":"stderr","time":"2017-09-09T17:54:54.712261219Z"} {"log":"E0909 17:57:34.716929 1 servicecontroller.go:753] Failed to process service. Retrying in 5m0s: Failed to create load balancer for service federation-system/e2e-federation-apiser ver: User \"system:kube-controller-manager\" cannot create configmaps in the namespace \"kube-system\". (post configmaps)\n","stream":"stderr","time":"2017-09-09T17:57:34.717073975Z"} — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[shashidharatd]

I digged into the code and found that the error is coming from gce cloud-provider. Here is the location https://github.com/kubernetes/kubernetes/blob/release-1.7/pkg/cloudprovider/providers/gce/gce_clusterid.go#L192

[shashidharatd]

@MrHohn, This issue is related to gce cloud-provider and RBAC role, rolebinding. Do you have any clue?

[luxas]

I think it's just ok to do a post-apply of these rules if you use GCE. With out-of-tree cloud providers, this will be in the same manifest anyway...

[MrHohn]

Seems we introduced new operations (for configmap) in gce controller but never handled its RBAC...
cc @nicksardo

Link:

{
	// a role to use for bootstrapping the kube-controller-manager so it can create the shared informers
	// service accounts, and secrets that we need to create separate identities for other controllers
	ObjectMeta: metav1.ObjectMeta{Name: "system:kube-controller-manager"},
	Rules: []rbac.PolicyRule{
		eventsRule(),
		rbac.NewRule("create").Groups(legacyGroup).Resources("endpoints", "secrets", "serviceaccounts").RuleOrDie(),
		rbac.NewRule("delete").Groups(legacyGroup).Resources("secrets").RuleOrDie(),
		rbac.NewRule("get").Groups(legacyGroup).Resources("endpoints", "namespaces", "secrets", "serviceaccounts").RuleOrDie(),
		rbac.NewRule("update").Groups(legacyGroup).Resources("endpoints", "secrets", "serviceaccounts").RuleOrDie(),
		// Needed to check API access.  These creates are non-mutating
		rbac.NewRule("create").Groups(authenticationGroup).Resources("tokenreviews").RuleOrDie(),
		// Needed for all shared informers
		rbac.NewRule("list", "watch").Groups("*").Resources("*").RuleOrDie(),
	},
},

[nicksardo]

Hmm. The policy was added here https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/auth/authorizer/rbac/bootstrappolicy/namespace_policy.go#L88-L94 and the client should be made with cloud-provider: https://github.com/kubernetes/kubernetes/blob/release-1.7/pkg/cloudprovider/providers/gce/gce_clusterid.go#L65

[shashidharatd]

/kind bug
/area cloudprovider
/assign @nicksardo