[occm] (helm chart) `cloud-controller-manager` serviceaccount has no clusterrolebinding #2049

euven · 2022-12-05T08:08:49Z

/kind bug

What happened:
Using --use-service-account-crredentials (as is default in the chart), I'm seeing the following in the logs:

I1202 01:58:51.386134       1 event.go:294] "Event occurred" object="test/nginx-ingress-nginx-ingress" fieldPath="" kind="Service" apiVersion="v1" type="Warning" reason="SyncLoadBalancerFailed" message="Error syncing load balancer: failed to ensure load balancer: failed to patch service object test/nginx-ingress-nginx-ingress: services \"nginx-ingress-nginx-ingress\" is forbidden: User \"system:serviceaccount:kube-system:cloud-controller-manager\" cannot patch resource \"services\" in API group \"\" in the namespace \"test\""

Should the roles stuff in the manifests folder be part of the chart, or am I misunderstanding how this works?

When I set --use-service-account-credentials=false via controllerExtraArgs, things seem to work as expected.

What you expected to happen:
No permission errors in the logs

How to reproduce it:
Install the helm chart with values:

    image:
      tag: v1.24.5

The text was updated successfully, but these errors were encountered:

jichenjc · 2022-12-06T02:42:22Z

I am wondering whether your namespace is incorrect or other manual things handled wrong

https://github.com/kubernetes/cloud-provider-openstack/blob/master/charts/openstack-cloud-controller-manager/templates/clusterrole.yaml#L36 is the role that defined and there is a rolebinding that helps grant the actions, maybe you can check whether there's something wrong there

the option you provided seems reltaed to
https://github.com/kubernetes/cloud-provider/blob/master/options/options.go#L187
so likely it will create a client that allow Anonym access which not something we desire to though it can solve your current issue

euven · 2022-12-07T21:42:26Z

Thanks for taking a look @jichenjc.

The namespace I'm installing this into is kube-system.

https://github.com/kubernetes/cloud-provider-openstack/blob/master/charts/openstack-cloud-controller-manager/templates/clusterrole.yaml#L36 is the role that defined and there is a rolebinding that helps grant the actions, maybe you can check whether there's something wrong there

Note the serviceaccount there is system:openstack-cloud-controller-manager, which does get used, and works, when you set --use-service-account-credentials=false.
The issue seems to arise when you set --use-service-account-credentials=true: system:serviceaccount:kube-system:cloud-controller-manager seems to be used then not system:openstack-cloud-controller-manager). While system:serviceaccount:kube-system:cloud-controller-manager does exist, there seems to be no clusterrolebinding, hence asking if the relevant things in manifests should be part of the chart, if --use-service-account-credentials=true is used?

jichenjc · 2022-12-08T03:53:02Z

ok, this reminds me #1722
--use-service-account-credentials=true seems talked there

can you help check this issue?

seems it's fixed #1755 but I didn't personally check
whether 1.24.5 still have (maybe forgot to backport?)

zifeo · 2022-12-10T00:57:45Z

@jichenjc Got same issue with 1.25.0, same solution. Isn't #1755 missing the openstack- part?

jichenjc · 2022-12-12T01:34:49Z

Got same issue with 1.25.0, same solution. Isn't #1755 missing the openstack- part?

um.. the fix is there for long time .. so if it's a 1.25 potential fix, then it's a new issue :(

namsral · 2023-01-05T16:03:47Z

Had a similar issue after updating the openstack-cloud-controller-manager chart from 1.3.0 to 1.4.0

New services backed by load-balancers would not get an external IP because the serviceaccount cloud-controller-manager did not have the proper privileges to patch the service object.

Error:

Warning  SyncLoadBalancerFailed  2m30s (x2 over 5m11s)  service-controller  Error syncing load balancer: failed to ensure load balancer: failed to patch service object istio-ingress/istio-ingressgateway: services "istio-ingressgateway" is forbidden: User "system:serviceaccount:kube-system:cloud-controller-manager" cannot patch resource "services" in API group "" in the namespace "istio-ingress"

My temporary workaround piggybacking on the clusterrole system:openstack-cloud-controller-manager:

% kubectl apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:cloud-controller-manager
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:openstack-cloud-controller-manager
subjects:
- kind: ServiceAccount
  name: cloud-controller-manager
  namespace: kube-system
EOF

yankcrime · 2023-01-27T19:37:51Z

We've started hitting consistently as well with v1.25.0 (since that's the latest version deployed via the Helm chart). The workaround suggested by @namsral does the trick.

lukasmrtvy · 2023-01-28T00:26:10Z

@namsral
the error comes from service-controller ( kube-controller-manager ) which is using "system:serviceaccount:kube-system:cloud-controller-manager" user.

Helm chart has slightly different ClusterRoleBinding and RBAC ( https://github.com/kubernetes/cloud-provider-openstack/blob/master/charts/openstack-cloud-controller-manager/templates/clusterrolebinding.yaml ) "system:serviceaccount:kube-system:openstack-cloud-controller-manager"

I am not familiar with how it is working with the external cloud provider. @jichenjc

k8s-triage-robot · 2023-04-28T01:02:49Z

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue as fresh with /remove-lifecycle stale
Close this issue with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

nikParasyr · 2023-04-28T06:40:09Z

/remove-lifecycle stale

The default setting of the openstack-cloud-controller-manager uses two different service accounts with differing permissions. This leads to issues, especially when creating load-balancer services, where the wrong serivce account is used and consequently not allowed to do the things required to actually make things work. This is a workaround that forces the openstack-cloud-controller-manager to just use one single service account, which makes things work again. Fixes: Error syncing load balancer: failed to ensure load balancer: failed to patch service object ingress-nginx/ingress-nginx-controller: services "ingress-nginx-controller" is forbidden: User "system:serviceaccount:kube-system:cloud-controller-manager" cannot patch resource "services" in API group "" in the namespace "ingress-nginx" References: - kubernetes/cloud-provider-openstack#2049 - kubernetes/cloud-provider-openstack#1722

…credentials (#412) The default setting of the openstack-cloud-controller-manager uses two different service accounts with differing permissions. This leads to issues, especially when creating load-balancer services, where the wrong serivce account is used and consequently not allowed to do the things required to actually make things work. This is a workaround that forces the openstack-cloud-controller-manager to just use one single service account, which makes things work again. Fixes: Error syncing load balancer: failed to ensure load balancer: failed to patch service object ingress-nginx/ingress-nginx-controller: services "ingress-nginx-controller" is forbidden: User "system:serviceaccount:kube-system:cloud-controller-manager" cannot patch resource "services" in API group "" in the namespace "ingress-nginx" References: - kubernetes/cloud-provider-openstack#2049 - kubernetes/cloud-provider-openstack#1722 Co-authored-by: Sebastian Lehne <sl@teuto.net>

This removes the `openstack-` prefix from the service account name used by the cloud-controller-manager. The change is motivated by a number of reasons, namely: - Creates suitable service accounts, cluster roles and cluster role bindings for `--use-service-accounts-credentials=true` - Normalises service account names in the helm chart and plain manifests - Adhere to naming conventions across external cloud controller managers for different clouds (e.g. AWS, GCP, ...) Specifically the first point deserves further details. Prior to this change, users who install the cloud controller manager with helm, would run into the following error when creating load balancers: ``` E0818 08:27:33.802407 11 controller.go:291] error processing service default/hello-bug (will retry): failed to ensure load balancer: failed to patch service object default/hello-bug: services "hello-bug" is forbidden: User "system:serviceaccount:kube-system:cloud-controller-manager" cannot patch resource "services" in API group "" in the namespace "default" ``` Which is due to the fact that the controller is running with the `cloud-controller-manager` service account because `--use-service-account-credentials` is set to `true` by default and the client is initialised with: ``` clientset := clientBuilder.ClientOrDie("cloud-controller-manager") ``` Whilst users can work around this by passing `--use-service-account-credentials=false`, the desired behaviour would be to install suitable RBAC in the first place. See: - https://kubernetes.io/docs/concepts/architecture/cloud-controller/ - https://kubernetes.io/docs/tasks/administer-cluster/running-cloud-controller/ - kubernetes#2049 - kubernetes#1722 - kubernetes#1755 Signed-off-by: Wolodja Wentland <wwentland@wavecon.de>

This removes the `openstack-` prefix from the service account name used by the cloud-controller-manager. The change is motivated by the following: - Create suitable service accounts, cluster roles and cluster role bindings for use with `--use-service-accounts-credentials=true` - Normalise service account names in the helm chart and plain manifests - Adhere to naming conventions across external cloud controller managers for different clouds (e.g. AWS, GCP, ...) Specifically the first point deserves further details. Prior to this change, users who install the cloud controller manager with helm, would run into the following error when creating load balancers: ``` E0818 08:27:33.802407 11 controller.go:291] error processing service default/hello-bug (will retry): failed to ensure load balancer: failed to patch service object default/hello-bug: services "hello-bug" is forbidden: User "system:serviceaccount:kube-system:cloud-controller-manager" cannot patch resource "services" in API group "" in the namespace "default" ``` Which is due to the fact that the controller is running with the `cloud-controller-manager` service account because `--use-service-account-credentials` is set to `true` by default and the client is initialised with: ``` clientset := clientBuilder.ClientOrDie("cloud-controller-manager") ``` Whilst users can work around this by passing `--use-service-account-credentials=false`, the desired behaviour would be to install suitable RBAC in the first place. See: - https://kubernetes.io/docs/concepts/architecture/cloud-controller/ - https://kubernetes.io/docs/tasks/administer-cluster/running-cloud-controller/ - kubernetes#2049 - kubernetes#1722 - kubernetes#1755 Signed-off-by: Wolodja Wentland <wwentland@wavecon.de>

This removes the `openstack-` prefix from the service account name used by the cloud-controller-manager and moves the default values into the values file. The change is motivated by the following: - Create suitable service accounts, cluster roles and cluster role bindings for use with `--use-service-accounts-credentials=true` - Normalise service account names in the helm chart and plain manifests - Adhere to naming conventions across external cloud controller managers for different clouds (e.g. AWS, GCP, ...) Specifically the first point deserves further details. Prior to this change, users who install the cloud controller manager with helm, would run into the following error when creating load balancers: ``` E0818 08:27:33.802407 11 controller.go:291] error processing service default/hello-bug (will retry): failed to ensure load balancer: failed to patch service object default/hello-bug: services "hello-bug" is forbidden: User "system:serviceaccount:kube-system:cloud-controller-manager" cannot patch resource "services" in API group "" in the namespace "default" ``` Which is due to the fact that the controller is running with the `cloud-controller-manager` service account because `--use-service-account-credentials` is set to `true` by default and the client is initialised with: ``` clientset := clientBuilder.ClientOrDie("cloud-controller-manager") ``` Whilst users can work around this by passing `--use-service-account-credentials=false`, the desired behaviour would be to install suitable RBAC in the first place. See: - https://kubernetes.io/docs/concepts/architecture/cloud-controller/ - https://kubernetes.io/docs/tasks/administer-cluster/running-cloud-controller/ - kubernetes#2049 - kubernetes#1722 - kubernetes#1755 Signed-off-by: Wolodja Wentland <wwentland@wavecon.de>

This removes the `openstack-` prefix from the service account name used by the cloud-controller-manager and moves the default values into the values file. The change is motivated by the following: - Create suitable service accounts, cluster roles and cluster role bindings for use with `--use-service-accounts-credentials=true` - Normalise service account names in the helm chart and plain manifests - Adhere to naming conventions across external cloud controller managers for different clouds (e.g. AWS, GCP, ...) Specifically the first point deserves further details. Prior to this change, users who install the cloud controller manager with helm, would run into the following error when creating load balancers: ``` E0818 08:27:33.802407 11 controller.go:291] error processing service default/hello-bug (will retry): failed to ensure load balancer: failed to patch service object default/hello-bug: services "hello-bug" is forbidden: User "system:serviceaccount:kube-system:cloud-controller-manager" cannot patch resource "services" in API group "" in the namespace "default" ``` Which is due to the fact that the controller is running with the `cloud-controller-manager` service account because `--use-service-account-credentials` is set to `true` by default and the client is initialised with: ``` clientset := clientBuilder.ClientOrDie("cloud-controller-manager") ``` Whilst users can work around this by passing `--use-service-account-credentials=false`, the desired behaviour would be to install suitable RBAC in the first place. See: - https://kubernetes.io/docs/concepts/architecture/cloud-controller/ - https://kubernetes.io/docs/tasks/administer-cluster/running-cloud-controller/ - #2049 - #1722 - #1755 Signed-off-by: Wolodja Wentland <wwentland@wavecon.de>

This removes the `openstack-` prefix from the service account name used by the cloud-controller-manager and moves the default values into the values file. The change is motivated by the following: - Create suitable service accounts, cluster roles and cluster role bindings for use with `--use-service-accounts-credentials=true` - Normalise service account names in the helm chart and plain manifests - Adhere to naming conventions across external cloud controller managers for different clouds (e.g. AWS, GCP, ...) Specifically the first point deserves further details. Prior to this change, users who install the cloud controller manager with helm, would run into the following error when creating load balancers: ``` E0818 08:27:33.802407 11 controller.go:291] error processing service default/hello-bug (will retry): failed to ensure load balancer: failed to patch service object default/hello-bug: services "hello-bug" is forbidden: User "system:serviceaccount:kube-system:cloud-controller-manager" cannot patch resource "services" in API group "" in the namespace "default" ``` Which is due to the fact that the controller is running with the `cloud-controller-manager` service account because `--use-service-account-credentials` is set to `true` by default and the client is initialised with: ``` clientset := clientBuilder.ClientOrDie("cloud-controller-manager") ``` Whilst users can work around this by passing `--use-service-account-credentials=false`, the desired behaviour would be to install suitable RBAC in the first place. See: - https://kubernetes.io/docs/concepts/architecture/cloud-controller/ - https://kubernetes.io/docs/tasks/administer-cluster/running-cloud-controller/ - kubernetes#2049 - kubernetes#1722 - kubernetes#1755 Signed-off-by: Wolodja Wentland <wwentland@wavecon.de>

…chart (#2449) * Use standard service account name in OCCM helm chart (#2332) This removes the `openstack-` prefix from the service account name used by the cloud-controller-manager and moves the default values into the values file. The change is motivated by the following: - Create suitable service accounts, cluster roles and cluster role bindings for use with `--use-service-accounts-credentials=true` - Normalise service account names in the helm chart and plain manifests - Adhere to naming conventions across external cloud controller managers for different clouds (e.g. AWS, GCP, ...) Specifically the first point deserves further details. Prior to this change, users who install the cloud controller manager with helm, would run into the following error when creating load balancers: ``` E0818 08:27:33.802407 11 controller.go:291] error processing service default/hello-bug (will retry): failed to ensure load balancer: failed to patch service object default/hello-bug: services "hello-bug" is forbidden: User "system:serviceaccount:kube-system:cloud-controller-manager" cannot patch resource "services" in API group "" in the namespace "default" ``` Which is due to the fact that the controller is running with the `cloud-controller-manager` service account because `--use-service-account-credentials` is set to `true` by default and the client is initialised with: ``` clientset := clientBuilder.ClientOrDie("cloud-controller-manager") ``` Whilst users can work around this by passing `--use-service-account-credentials=false`, the desired behaviour would be to install suitable RBAC in the first place. See: - https://kubernetes.io/docs/concepts/architecture/cloud-controller/ - https://kubernetes.io/docs/tasks/administer-cluster/running-cloud-controller/ - #2049 - #1722 - #1755 Signed-off-by: Wolodja Wentland <wwentland@wavecon.de> * Remove spurious whitespace in OCCM values file (#2347) The space character was sadly introduced in an earlier PR and had not been picked up by the helm linter, presumably because the GHA did not run. Signed-off-by: Wolodja Wentland <w@wentland.dev> --------- Signed-off-by: Wolodja Wentland <wwentland@wavecon.de> Signed-off-by: Wolodja Wentland <w@wentland.dev>

…chart (#2448) * Use standard service account name in OCCM helm chart (#2332) This removes the `openstack-` prefix from the service account name used by the cloud-controller-manager and moves the default values into the values file. The change is motivated by the following: - Create suitable service accounts, cluster roles and cluster role bindings for use with `--use-service-accounts-credentials=true` - Normalise service account names in the helm chart and plain manifests - Adhere to naming conventions across external cloud controller managers for different clouds (e.g. AWS, GCP, ...) Specifically the first point deserves further details. Prior to this change, users who install the cloud controller manager with helm, would run into the following error when creating load balancers: ``` E0818 08:27:33.802407 11 controller.go:291] error processing service default/hello-bug (will retry): failed to ensure load balancer: failed to patch service object default/hello-bug: services "hello-bug" is forbidden: User "system:serviceaccount:kube-system:cloud-controller-manager" cannot patch resource "services" in API group "" in the namespace "default" ``` Which is due to the fact that the controller is running with the `cloud-controller-manager` service account because `--use-service-account-credentials` is set to `true` by default and the client is initialised with: ``` clientset := clientBuilder.ClientOrDie("cloud-controller-manager") ``` Whilst users can work around this by passing `--use-service-account-credentials=false`, the desired behaviour would be to install suitable RBAC in the first place. See: - https://kubernetes.io/docs/concepts/architecture/cloud-controller/ - https://kubernetes.io/docs/tasks/administer-cluster/running-cloud-controller/ - #2049 - #1722 - #1755 Signed-off-by: Wolodja Wentland <wwentland@wavecon.de> * Remove spurious whitespace in OCCM values file (#2347) The space character was sadly introduced in an earlier PR and had not been picked up by the helm linter, presumably because the GHA did not run. Signed-off-by: Wolodja Wentland <w@wentland.dev> --------- Signed-off-by: Wolodja Wentland <wwentland@wavecon.de> Signed-off-by: Wolodja Wentland <w@wentland.dev>

…chart (#2447) * Use standard service account name in OCCM helm chart (#2332) This removes the `openstack-` prefix from the service account name used by the cloud-controller-manager and moves the default values into the values file. The change is motivated by the following: - Create suitable service accounts, cluster roles and cluster role bindings for use with `--use-service-accounts-credentials=true` - Normalise service account names in the helm chart and plain manifests - Adhere to naming conventions across external cloud controller managers for different clouds (e.g. AWS, GCP, ...) Specifically the first point deserves further details. Prior to this change, users who install the cloud controller manager with helm, would run into the following error when creating load balancers: ``` E0818 08:27:33.802407 11 controller.go:291] error processing service default/hello-bug (will retry): failed to ensure load balancer: failed to patch service object default/hello-bug: services "hello-bug" is forbidden: User "system:serviceaccount:kube-system:cloud-controller-manager" cannot patch resource "services" in API group "" in the namespace "default" ``` Which is due to the fact that the controller is running with the `cloud-controller-manager` service account because `--use-service-account-credentials` is set to `true` by default and the client is initialised with: ``` clientset := clientBuilder.ClientOrDie("cloud-controller-manager") ``` Whilst users can work around this by passing `--use-service-account-credentials=false`, the desired behaviour would be to install suitable RBAC in the first place. See: - https://kubernetes.io/docs/concepts/architecture/cloud-controller/ - https://kubernetes.io/docs/tasks/administer-cluster/running-cloud-controller/ - #2049 - #1722 - #1755 Signed-off-by: Wolodja Wentland <wwentland@wavecon.de> * Remove spurious whitespace in OCCM values file (#2347) The space character was sadly introduced in an earlier PR and had not been picked up by the helm linter, presumably because the GHA did not run. Signed-off-by: Wolodja Wentland <w@wentland.dev> --------- Signed-off-by: Wolodja Wentland <wwentland@wavecon.de> Signed-off-by: Wolodja Wentland <w@wentland.dev>

k8s-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Dec 5, 2022

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Apr 28, 2023

k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Apr 28, 2023

ghost mentioned this issue May 10, 2023

fix(t8s-cluster/cloud-controller-manager): Don't use service-account credentials teutonet/teutonet-helm-charts#412

Merged

dhrp mentioned this issue Aug 21, 2023

added a temporary? patch for the CCM leafcloudhq/terraform-examples#5

Open

wwentland mentioned this issue Aug 22, 2023

[occm] Use standard service account name in OCCM helm chart #2332

Merged

k8s-ci-robot closed this as completed in #2332 Aug 28, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[occm] (helm chart) `cloud-controller-manager` serviceaccount has no clusterrolebinding #2049

[occm] (helm chart) `cloud-controller-manager` serviceaccount has no clusterrolebinding #2049

euven commented Dec 5, 2022 •

edited

jichenjc commented Dec 6, 2022

euven commented Dec 7, 2022

jichenjc commented Dec 8, 2022 •

edited

zifeo commented Dec 10, 2022

jichenjc commented Dec 12, 2022

namsral commented Jan 5, 2023 •

edited

yankcrime commented Jan 27, 2023

lukasmrtvy commented Jan 28, 2023

k8s-triage-robot commented Apr 28, 2023

nikParasyr commented Apr 28, 2023

[occm] (helm chart) cloud-controller-manager serviceaccount has no clusterrolebinding #2049

[occm] (helm chart) cloud-controller-manager serviceaccount has no clusterrolebinding #2049

Comments

euven commented Dec 5, 2022 • edited

jichenjc commented Dec 6, 2022

euven commented Dec 7, 2022

jichenjc commented Dec 8, 2022 • edited

zifeo commented Dec 10, 2022

jichenjc commented Dec 12, 2022

namsral commented Jan 5, 2023 • edited

yankcrime commented Jan 27, 2023

lukasmrtvy commented Jan 28, 2023

k8s-triage-robot commented Apr 28, 2023

nikParasyr commented Apr 28, 2023

[occm] (helm chart) `cloud-controller-manager` serviceaccount has no clusterrolebinding #2049

[occm] (helm chart) `cloud-controller-manager` serviceaccount has no clusterrolebinding #2049

euven commented Dec 5, 2022 •

edited

jichenjc commented Dec 8, 2022 •

edited

namsral commented Jan 5, 2023 •

edited