Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unable to install Keda 2.7 operator with securityContext added #3012

Closed
aslom opened this issue May 5, 2022 · 5 comments · Fixed by #3015
Closed

Unable to install Keda 2.7 operator with securityContext added #3012

aslom opened this issue May 5, 2022 · 5 comments · Fixed by #3015
Assignees
@zroubalik
Labels
bug Something isn't working
Milestone
v2.7.1

Comments

@aslom
Copy link

aslom commented May 5, 2022 •
edited

Report

Unable to install Keda 2.7 operator with seurity securityContext added

Expected Behavior

With previous version:

 kubectl apply -f https://github.com/kedacore/keda/releases/download/v2.6.1/keda-2.6.1.yaml
namespace/keda configured
customresourcedefinition.apiextensions.k8s.io/clustertriggerauthentications.keda.sh configured
customresourcedefinition.apiextensions.k8s.io/scaledjobs.keda.sh configured
customresourcedefinition.apiextensions.k8s.io/scaledobjects.keda.sh configured
customresourcedefinition.apiextensions.k8s.io/triggerauthentications.keda.sh configured
serviceaccount/keda-operator configured
clusterrole.rbac.authorization.k8s.io/keda-external-metrics-reader configured
clusterrole.rbac.authorization.k8s.io/keda-operator configured
rolebinding.rbac.authorization.k8s.io/keda-auth-reader configured
clusterrolebinding.rbac.authorization.k8s.io/keda-hpa-controller-external-metrics configured
clusterrolebinding.rbac.authorization.k8s.io/keda-operator configured
clusterrolebinding.rbac.authorization.k8s.io/keda-system-auth-delegator configured
service/keda-metrics-apiserver configured
deployment.apps/keda-metrics-apiserver configured
deployment.apps/keda-operator configured
apiservice.apiregistration.k8s.io/v1beta1.external.metrics.k8s.io configured

and both pods get started as expected:

aslom@m:~/Documents/awsm/go/src/github.com/kedacore/keda|main⚡ ⇒  
k -n keda get po
NAME                                      READY   STATUS    RESTARTS   AGE
keda-metrics-apiserver-59b9ddc78c-8bj8c   1/1     Running   0          41h
keda-operator-f76d844d7-6c27x             1/1     Running   0          15h

k -n keda get deployment keda-operator -oyaml
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "2"
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"app":"keda-operator","app.kubernetes.io/component":"operator","app.kubernetes.io/name":"keda-operator","app.kubernetes.io/part-of":"keda-operator","app.kubernetes.io/version":"2.6.1"},"name":"keda-operator","namespace":"keda"},"spec":{"replicas":1,"selector":{"matchLabels":{"app":"keda-operator"}},"template":{"metadata":{"labels":{"app":"keda-operator","name":"keda-operator"},"name":"keda-operator"},"spec":{"containers":[{"args":["--leader-elect","--zap-log-level=info","--zap-encoder=console"],"command":["/keda"],"env":[{"name":"WATCH_NAMESPACE","value":""},{"name":"KEDA_HTTP_DEFAULT_TIMEOUT","value":""}],"image":"ghcr.io/kedacore/keda:2.6.1","imagePullPolicy":"Always","livenessProbe":{"httpGet":{"path":"/healthz","port":8081},"initialDelaySeconds":25},"name":"keda-operator","ports":[{"containerPort":8080,"name":"http","protocol":"TCP"}],"readinessProbe":{"httpGet":{"path":"/readyz","port":8081},"initialDelaySeconds":20},"resources":{"limits":{"cpu":"1000m","memory":"1000Mi"},"requests":{"cpu":"100m","memory":"100Mi"}}}],"nodeSelector":{"kubernetes.io/os":"linux"},"serviceAccountName":"keda-operator","terminationGracePeriodSeconds":10}}}}
  creationTimestamp: "2022-05-05T20:34:50Z"
  generation: 2
  labels:
    app: keda-operator
    app.kubernetes.io/component: operator
    app.kubernetes.io/name: keda-operator
    app.kubernetes.io/part-of: keda-operator
    app.kubernetes.io/version: 2.6.1
  name: keda-operator
  namespace: keda
  resourceVersion: "788722238"
  uid: a5d46ff3-4ae0-4a8f-a29e-b4c63d056e1b
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: keda-operator
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: keda-operator
        name: keda-operator
      name: keda-operator
    spec:
      containers:
      - args:
        - --leader-elect
        - --zap-log-level=info
        - --zap-encoder=console
        command:
        - /keda
        env:
        - name: WATCH_NAMESPACE
        - name: KEDA_HTTP_DEFAULT_TIMEOUT
        image: ghcr.io/kedacore/keda:2.6.1
        imagePullPolicy: Always
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 8081
            scheme: HTTP
          initialDelaySeconds: 25
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: keda-operator
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /readyz
            port: 8081
            scheme: HTTP
          initialDelaySeconds: 20
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources:
          limits:
            cpu: "1"
            memory: 1000Mi
          requests:
            cpu: 100m
            memory: 100Mi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      nodeSelector:
        kubernetes.io/os: linux
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: keda-operator
      serviceAccountName: keda-operator
      terminationGracePeriodSeconds: 10
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2022-05-05T20:37:33Z"
    lastUpdateTime: "2022-05-05T20:37:33Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  - lastTransitionTime: "2022-05-05T20:34:50Z"
    lastUpdateTime: "2022-05-05T20:37:33Z"
    message: ReplicaSet "keda-operator-f76d844d7" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  observedGeneration: 2
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

Actual Behavior

kubectl apply -f https://github.com/kedacore/keda/releases/download/v2.7.0/keda-2.7.0.yaml
namespace/keda unchanged
customresourcedefinition.apiextensions.k8s.io/clustertriggerauthentications.keda.sh configured
customresourcedefinition.apiextensions.k8s.io/scaledjobs.keda.sh configured
customresourcedefinition.apiextensions.k8s.io/scaledobjects.keda.sh configured
customresourcedefinition.apiextensions.k8s.io/triggerauthentications.keda.sh configured
serviceaccount/keda-operator unchanged
clusterrole.rbac.authorization.k8s.io/keda-external-metrics-reader unchanged
clusterrole.rbac.authorization.k8s.io/keda-operator configured
rolebinding.rbac.authorization.k8s.io/keda-auth-reader unchanged
clusterrolebinding.rbac.authorization.k8s.io/keda-hpa-controller-external-metrics unchanged
clusterrolebinding.rbac.authorization.k8s.io/keda-operator unchanged
clusterrolebinding.rbac.authorization.k8s.io/keda-system-auth-delegator unchanged
service/keda-metrics-apiserver unchanged
deployment.apps/keda-metrics-apiserver configured
deployment.apps/keda-operator created
apiservice.apiregistration.k8s.io/v1beta1.external.metrics.k8s.io unchanged

Operator pod not available:

k -n keda get po
NAME                                      READY   STATUS    RESTARTS   AGE
keda-metrics-apiserver-59b9ddc78c-8bj8c   1/1     Running   0          25h

k -n keda get deployment keda-operator  -oyaml
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"app":"keda-operator","app.kubernetes.io/component":"operator","app.kubernetes.io/name":"keda-operator","app.kubernetes.io/part-of":"keda-operator","app.kubernetes.io/version":"2.7.0"},"name":"keda-operator","namespace":"keda"},"spec":{"replicas":1,"selector":{"matchLabels":{"app":"keda-operator"}},"template":{"metadata":{"labels":{"app":"keda-operator","name":"keda-operator"},"name":"keda-operator"},"spec":{"containers":[{"args":["--leader-elect","--zap-log-level=info","--zap-encoder=console"],"command":["/keda"],"env":[{"name":"WATCH_NAMESPACE","value":""},{"name":"KEDA_HTTP_DEFAULT_TIMEOUT","value":""}],"image":"ghcr.io/kedacore/keda:2.7.0","imagePullPolicy":"Always","livenessProbe":{"httpGet":{"path":"/healthz","port":8081},"initialDelaySeconds":25},"name":"keda-operator","ports":[{"containerPort":8080,"name":"http","protocol":"TCP"}],"readinessProbe":{"httpGet":{"path":"/readyz","port":8081},"initialDelaySeconds":20},"resources":{"limits":{"cpu":"1000m","memory":"1000Mi"},"requests":{"cpu":"100m","memory":"100Mi"}},"securityContext":{"allowPrivilegeEscalation":false,"capabilities":{"drop":["ALL"]},"readOnlyRootFilesystem":true}}],"nodeSelector":{"kubernetes.io/os":"linux"},"securityContext":{"fsGroup":1000,"runAsGroup":1000,"runAsNonRoot":true,"runAsUser":1000},"serviceAccountName":"keda-operator","terminationGracePeriodSeconds":10}}}}
  creationTimestamp: "2022-05-05T20:34:50Z"
  generation: 1
  labels:
    app: keda-operator
    app.kubernetes.io/component: operator
    app.kubernetes.io/name: keda-operator
    app.kubernetes.io/part-of: keda-operator
    app.kubernetes.io/version: 2.7.0
  name: keda-operator
  namespace: keda
  resourceVersion: "788717063"
  uid: a5d46ff3-4ae0-4a8f-a29e-b4c63d056e1b
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: keda-operator
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: keda-operator
        name: keda-operator
      name: keda-operator
    spec:
      containers:
      - args:
        - --leader-elect
        - --zap-log-level=info
        - --zap-encoder=console
        command:
        - /keda
        env:
        - name: WATCH_NAMESPACE
        - name: KEDA_HTTP_DEFAULT_TIMEOUT
        image: ghcr.io/kedacore/keda:2.7.0
        imagePullPolicy: Always
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 8081
            scheme: HTTP
          initialDelaySeconds: 25
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: keda-operator
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /readyz
            port: 8081
            scheme: HTTP
          initialDelaySeconds: 20
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources:
          limits:
            cpu: "1"
            memory: 1000Mi
          requests:
            cpu: 100m
            memory: 100Mi
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          readOnlyRootFilesystem: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      nodeSelector:
        kubernetes.io/os: linux
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 1000
        runAsGroup: 1000
        runAsNonRoot: true
        runAsUser: 1000
      serviceAccount: keda-operator
      serviceAccountName: keda-operator
      terminationGracePeriodSeconds: 10
status:
  conditions:
  - lastTransitionTime: "2022-05-05T20:34:50Z"
    lastUpdateTime: "2022-05-05T20:34:50Z"
    message: Created new replica set "keda-operator-858d6f8878"
    reason: NewReplicaSetCreated
    status: "True"
    type: Progressing
  - lastTransitionTime: "2022-05-05T20:34:50Z"
    lastUpdateTime: "2022-05-05T20:34:50Z"
    message: Deployment does not have minimum availability.
    reason: MinimumReplicasUnavailable
    status: "False"
    type: Available
  - lastTransitionTime: "2022-05-05T20:34:50Z"
    lastUpdateTime: "2022-05-05T20:34:50Z"
    message: 'pods "keda-operator-858d6f8878-" is forbidden: unable to validate against
      any security context constraint: [provider "anyuid": Forbidden: not usable by
      user or serviceaccount, provider restricted: .spec.securityContext.fsGroup:
      Invalid value: []int64{1000}: 1000 is not an allowed group, spec.containers[0].securityContext.runAsUser:
      Invalid value: 1000: must be in the ranges: [1000710000, 1000719999], provider
      "ibm-restricted-scc": Forbidden: not usable by user or serviceaccount, provider
      "nonroot": Forbidden: not usable by user or serviceaccount, provider "ibm-anyuid-scc":
      Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid":
      Forbidden: not usable by user or serviceaccount, provider "ibm-anyuid-hostpath-scc":
      Forbidden: not usable by user or serviceaccount, provider "machine-api-termination-handler":
      Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden:
      not usable by user or serviceaccount, provider "hostaccess": Forbidden: not
      usable by user or serviceaccount, provider "ibm-anyuid-hostaccess-scc": Forbidden:
      not usable by user or serviceaccount, provider "node-exporter": Forbidden: not
      usable by user or serviceaccount, provider "ibm-privileged-scc": Forbidden:
      not usable by user or serviceaccount, provider "privileged": Forbidden: not
      usable by user or serviceaccount]'
    reason: FailedCreate
    status: "True"
    type: ReplicaFailure
  observedGeneration: 1
  unavailableReplicas: 1

Steps to Reproduce the Problem

  1. Try to install Keda 2.7 in OpenShft 4.9
  2. Try to install Keda 2.6.1 in OpenShft 4.9

Logs from KEDA operator

N/A

KEDA Version

2.7.0

Kubernetes Version

1.22

Platform

Red Hat OpenShift

Scaler Details

n/A

Anything else?

This looks like regression introduced in
#2938
related issue
#2933
@aslom aslom added the bug Something isn't working label May 5, 2022
@tomkerkhove
Copy link
Member

Did it pick up because if there would be an issue then it should go in crashbackloop? What do the operator logs say?

This was referenced May 6, 2022
Merged
Merged
@aslom
Copy link
Author

aslom commented May 6, 2022 •
edited

@tomkerkhove no operator log as its deployment does not work - operator is prevented from starting AFAICT (I updated description to show that for 2.6.1 both pods are started and for 2.7 only one pod is runing)

@zroubalik zroubalik self-assigned this May 6, 2022
@tomkerkhove
Copy link
Member

Ah this is OpenShift, @zroubalik is already working on a fix in #3015

@aslom aslom changed the title Unable to install Keda 2.7 operator with seurity securityContext added Unable to install Keda 2.7 operator with securityContext added May 6, 2022
@tomkerkhove tomkerkhove added this to the v2.7.1 milestone May 6, 2022
@t0rr3sp3dr0 t0rr3sp3dr0 mentioned this issue May 12, 2023
Merged
3 tasks
@edubois10
Copy link

edubois10 commented Sep 21, 2023 •
edited

Hi there,

I have the same issue than @aslom but for the version v2.11.2

I do the same oc apply -f https://github.com/kedacore/keda/releases/download/v2.11.2/keda-2.11.2.yaml and I get the same scc errors.

I am as well on OpenShift

@edubois10
Copy link

Actually the issue appears again already in the v2.9.1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@zroubalik zroubalik

Labels
bug Something isn't working
Projects
Roadmap - KEDA Core
Status: Done
Milestone
v2.7.1
Development

Successfully merging a pull request may close this issue.

Don't hardcode UIDs in securityContext zroubalik/keda
4 participants
@zroubalik @aslom @tomkerkhove @edubois10