[openstack-cloud-controller-manager] Serviceaccount cannot patch resource "services"

[jichenjc]

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:

/kind bug
/kind feature

What happened:

create a LB follow
https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/openstack-cloud-controller-manager/using-openstack-cloud-controller-manager.md

kind: Service
apiVersion: v1
metadata:
  name: hello-world
  annotations:
    service.beta.kubernetes.io/brightbox-load-balancer-healthcheck-request: /
spec:
  type: LoadBalancer
  selector:
    app: hello-world
  ports:
    - name: http
      protocol: TCP
      port: 80
      targetPort: web

in logs seems

E1226 09:11:24.522851       1 controller.go:307] error processing service example/hello-world (will retry): failed to ensure load balancer: failed to patch service object example/hello-world: services "hello-world" is forbidden: User "system:serviceaccount:kube-system:cloud-provider-openstack" cannot patch resource "services" in API group "" in the namespace "example"
I1226 09:11:24.523070       1 event.go:291] "Event occurred" object="example/hello-world" kind="Service" apiVersion="v1" type="Warning" reason="SyncLoadBalancerFailed" message="Error syncing load balancer: failed to ensure load balancer: failed to patch service object example/hello-world: services \"hello-world\" is forbidden: User \"system:serviceaccount:kube-system:cloud-provider-openstack\" cannot patch resource \"services\" in API group \"\" in the namespace \"example\""

it's weird why "system:serviceaccount:kube-system:cloud-provider-openstack\" is used

I created following command

kubectl create clusterrolebinding my-1 --clusterrole=system:cloud-controller-manager --serviceaccount=kube-system:cloud-provider-openstack

and it works fine in the LB after the clusterrolebinding created
it's weird why kube-system:cloud-provider-openstack is created
What you expected to happen:

How to reproduce it:

Anything else we need to know?:

Environment:

• openstack-cloud-controller-manager(or other related binary) version:
• OpenStack version:
• Others:

[jichenjc]

opening this to track the problem as this is first time I saw this kind of issue

[jacksgt]

Hello,

I'm still seeing this error with the release-1.23 version (0807b16) and the latest Helm chart from master (d0f3cad).

kubectl create svc loadbalancer mv-lb results in:

E0224 09:41:12.619482       1 controller.go:310] error processing service default/mv-lb (will retry): failed to ensure load balancer: failed to patch service object default/mv-lb: services "mv-lb" is forbidden: User "system:serviceaccount:kube-system:cloud-controller-manager" cannot patch resource "services" in API group "" in the namespace "default"

The daemonset is using the service account openstack-cloud-controller-manager:

  spec:
    revisionHistoryLimit: 10
    selector:
      matchLabels:
        app: openstack-cloud-controller-manager
        component: controllermanager
        release: openstack-cloud-controller
    template:
      metadata:
        annotations:
          checksum/config: ed0a9897da2498221063e2119422aafa31b3f2f255035fcdf6774b8f22dd32fd
          kubectl.kubernetes.io/restartedAt: "2022-02-24T10:39:33+01:00"
        creationTimestamp: null
        labels:
          app: openstack-cloud-controller-manager
          chart: openstack-cloud-controller-manager-1.2.0
          component: controllermanager
          heritage: Helm
          release: openstack-cloud-controller
      spec:
        containers:
        - args:
          - /bin/openstack-cloud-controller-manager
          - --v=2
          - --cloud-config=$(CLOUD_CONFIG)
          - --cloud-provider=openstack
          - --use-service-account-credentials=true
          - --controllers=service
          - --bind-address=0.0.0.0
          - --cluster-name=clu-jack-dev
          env:
          - name: CLOUD_CONFIG
            value: /etc/config/cloud.conf
          image: registry.cern.ch/jhensche/occm:cern-1.23-dev
          imagePullPolicy: IfNotPresent
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: http
              scheme: HTTPS
            initialDelaySeconds: 60
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          name: openstack-cloud-controller-manager
          ports:
          - containerPort: 10258
            hostPort: 10258
            name: http
            protocol: TCP
          resources:
            limits:
              memory: 512Mi
            requests:
              memory: 50Mi
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
          - mountPath: /etc/config
            name: cloud-config-volume
            readOnly: true
        dnsPolicy: ClusterFirst
        hostNetwork: true
        nodeSelector:
          node-role.kubernetes.io/master: ""
        restartPolicy: Always
        schedulerName: default-scheduler
        securityContext: {}
        serviceAccount: openstack-cloud-controller-manager
        serviceAccountName: openstack-cloud-controller-manager
        terminationGracePeriodSeconds: 30
        tolerations:
        - effect: NoSchedule
          key: node.cloudprovider.kubernetes.io/uninitialized
          value: "true"
        - effect: NoSchedule
          key: node-role.kubernetes.io/master
        volumes:
        - name: cloud-config-volume
          secret:
            defaultMode: 420
            secretName: cloud-config
    updateStrategy:
      rollingUpdate:
        maxSurge: 0
        maxUnavailable: 1
      type: RollingUpdate

which has the following ClusterRole / ClusterRoleBinding:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
  creationTimestamp: "2022-02-02T10:40:55Z"
  labels:
    argocd.argoproj.io/instance: openstack-cloud-controller
  name: system:openstack-cloud-controller-manager
  resourceVersion: "21118211"
  uid: eff6fcf3-cd16-45fe-8aab-b65805e92b01
rules:
- apiGroups:
  - coordination.k8s.io
  resources:
  - leases
  verbs:
  - get
  - create
  - update
- apiGroups:
  - ""
  resources:
  - events
  verbs:
  - create
  - patch
  - update
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - '*'
- apiGroups:
  - ""
  resources:
  - nodes/status
  verbs:
  - patch
- apiGroups:
  - ""
  resources:
  - services
  verbs:
  - list
  - patch
  - update
  - watch
- apiGroups:
  - ""
  resources:
  - serviceaccounts/token
  verbs:
  - create
- apiGroups:
  - ""
  resources:
  - serviceaccounts
  verbs:
  - create
  - get
- apiGroups:
  - ""
  resources:
  - persistentvolumes
  verbs:
  - '*'
- apiGroups:
  - ""
  resources:
  - endpoints
  verbs:
  - create
  - get
  - list
  - watch
  - update
- apiGroups:
  - ""
  resources:
  - configmaps
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - secrets
  verbs:
  - list
  - get
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  creationTimestamp: "2022-02-02T10:40:55Z"
  labels:
    argocd.argoproj.io/instance: openstack-cloud-controller
  name: system:openstack-cloud-controller-manager
  resourceVersion: "21118235"
  uid: b004e774-5ff8-4391-88e4-ccf39e231a8d
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:openstack-cloud-controller-manager
subjects:
- kind: ServiceAccount
  name: openstack-cloud-controller-manager
  namespace: openshift-cern-cloud-controller

In particular the ClusterRole has this part, so it should be allowed to modify services in all namespaces:

- apiGroups:
  - ""
  resources:
  - services
  verbs:
  - list
  - patch
  - update
  - watch

Since the controller is running witht the service account "openstack-cloud-controller-manager", I don't understand why it identifies itself as "system:serviceaccount:kube-system:cloud-controller-manager" (especially since it's NOT running in the kube-system namespace).

/ping @lingxiankong @jichenjc

[lingxiankong]

@jacksgt Which namespace is the openstack-cloud-controller-manager deployed?

[jacksgt]

Hi, it's deployed in the `openshift-cern-cloud-controller`. OCCM V1.20 was also deployed in the same namespace. On 26 February 2022 23:56:40 CET, Lingxian Kong ***@***.***> wrote: ***@***.*** Which namespace is the openstack-cloud-controller-manager deployed?

…

-- Reply to this email directly or view it on GitHub: #1722 (comment) You are receiving this because you were mentioned. Message ID: ***@***.***>

[lingxiankong]

Since the --use-service-account-credentials=true is specified, cloud controller manager will manage all the controllers' service account in kube-system namespace.

[jacksgt]

Hi Lingxian,
I don't understand what you are referring to. Could you elaborate?

To my understanding, --use-service-account-credentials=true means that OCCM will use the credentials of the pod it is running in - in my case this is the openstack-cloud-controller-manager service account in the openshift-cern-cloud-controller namespace.
This has been working like this in v1.20 of OCCM.

Why would OCCM now use a different service account (in a different namespace). Is this a breaking change / regression?

[lingxiankong]

Why would OCCM now use a different service account (in a different namespace). Is this a breaking change / regression?

I remember this change was added together with the shared load balancer feature support. However, keep in mind that setting the --use-service-account-credentials to true runs each control loop within the cloud controller manager using a separate service account credential. You didn't see the issue before because OCCM was not updating resources in its loop.

You can try to set --use-service-account-credentials=false, the cloud controller manager will use openstack-cloud-controller-manager service account in the openshift-cern-cloud-controller namespace.

[jacksgt]

Thank you for the explanation! I understood now.

As you suggested, I tried setting --use-service-account-credentials=false and it mostly worked, except this error:

failed to add load balancer cleanup finalizer: services "my-lb" is forbidden: User "system:serviceaccount:openshift-cern-cloud-controller:openstack-cloud-controller-manager" cannot patch resource "services/status" in API group "" in the namespace "default"

I had to slightly modify the Helm chart for the required permissions and submitted a PR here: #1802

[mkjpryor]

@jacksgt

How are you setting --use-service-account-credentials=false in the Helm chart?

It seems like it is hard-coded in the Helm chart here: https://github.com/kubernetes/cloud-provider-openstack/blob/master/charts/openstack-cloud-controller-manager/templates/daemonset.yaml#L43

[jacksgt]

@mkjpryor You can specify additional arguments with Values.controllerExtraArgs.
Since those will be added later on, they will overwrite the "hardcoded" arguments.

controllerExtraArgs:
  - "--use-service-account-credentials=false"

[till]

@jacksgt sorry for reviving this, but is this released?

I just checked an older cluster where we used the manifests (and kustomize) to deploy the OCCM. It seems like --use-service-account-credentials=true works there. But when I deploy the OCCM with the helm chart, I need to disable it?

Error message:

Error syncing load balancer: failed to ensure load balancer: failed to patch service object haproxy-ingress/haproxy-ingress: services "haproxy-ingress" is forbidden: User "system:serviceaccount:kube-system:cloud-controller-manager" cannot patch resource "services" in API group "" in the namespace "haproxy-ingress"

kubectl describe ClusterRole/system:openstack-cloud-controller-manager
Name:         system:openstack-cloud-controller-manager
Labels:       app.kubernetes.io/managed-by=Helm
Annotations:  meta.helm.sh/release-name: openstack-cloud-controller-manager
              meta.helm.sh/release-namespace: kube-system
PolicyRule:
  Resources                   Non-Resource URLs  Resource Names  Verbs
  ---------                   -----------------  --------------  -----
  nodes                       []                 []              [*]
  persistentvolumes           []                 []              [*]
  endpoints                   []                 []              [create get list watch update]
  serviceaccounts             []                 []              [create get]
  events                      []                 []              [create patch update]
  serviceaccounts/token       []                 []              [create]
  leases.coordination.k8s.io  []                 []              [get create update]
  configmaps                  []                 []              [get list watch]
  secrets                     []                 []              [list get watch]
  services                    []                 []              [list patch update watch]
  nodes/status                []                 []              [patch]
  services/status             []                 []              [patch]

It seems like it's assigned too:

kubectl get clusterrolebinding|grep openstack
system:openstack-cloud-controller-manager              ClusterRole/system:openstack-cloud-controller-manager                              8h

[till]

I deleted the cloud-controller-manager service account as I couldn't figure out where it came from.

It seems like the helm chart creates openstack-cloud-controller-manager, but the pod creates cloud-controller-manager. And that SA is used to patch the object, but the SA does not have the ClusterRoleBinding assigned.

[jacksgt]

Hi @till ,

I'm also no expert on this, but I think the flag is simply named very confusingly.
The behavior that I have observed is this:

• --use-service-account-credentials=true => uses the cloud-controller-manager serviceaccount in the kube-system namespace (I believe these credentials are forwarded through the Kubernetes API calls made to the CCM?)
• --use-service-account-credentials=false => uses the local credentials ($KUBECONFIG) or in the case of running inside kubernetes it uses the credentials of the spec.serviceAccountName (the serviceaccount token must be mounted into the pod)

I hope this helps.

[till]

@jacksgt cheers, and thanks for responding.

I think something with RBAC is broken but I created another ticket and there's already a PR to pick commits for a new release.