When adding nodeSelector and tolerations to schedule onto a specific node, secrets can no longer be fetched.

[adegoodyer]

For context I'm deploying Sonarqube which requires it's own node as it's resource heavy. That StatefulSet manifest is pretty large so I've stripped out the essential parts into a simple pod manifest that I've used to test with and to demonstrate the issue here.

---
apiVersion: v1
kind: Pod
metadata:
  name: nginx-basic-pod
  namespace: nginx-basic
  labels:
    app: nginx-basic
spec:
  serviceAccountName: sonarqube-service-account
  containers:
    - name: nginx
      image: nginx:1.25.0
      ports:
        - containerPort: 80
          name: http-web-svc
      env:
        - name: POSTGRES_POSTGRES_PASSWORD
          valueFrom:
            secretKeyRef:
              name: sonarqube-secret
              key: postgresql_postgres_password
        - name: POSTGRES_USER
          value: "sonarUser"
        - name: POSTGRES_PASSWORD
          valueFrom:
            secretKeyRef:
              name: sonarqube-secret
              key: postgresql_password
      volumeMounts:
        - name: secrets
          mountPath: "/mnt/secrets"
          readOnly: true
  volumes:
    - name: secrets
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: sonarqube-secret-provider-class

When I add the nodeSelector and tolerations the secret is no longer able to be retrieved and the Secret isn't created in the cluster..

---
apiVersion: v1
kind: Pod
metadata:
  name: nginx-basic-pod
  namespace: nginx-basic
  labels:
    app: nginx-basic
spec:
  serviceAccountName: sonarqube-service-account
  nodeSelector:
    sonarqube: "true"
  tolerations:
    - effect: NoSchedule
      key: sonarqube
      operator: Equal
      value: "true"
  containers:
    - name: nginx
      image: nginx:1.25.0
      ports:
        - containerPort: 80
          name: http-web-svc
      env:
        - name: POSTGRES_POSTGRES_PASSWORD
          valueFrom:
            secretKeyRef:
              name: sonarqube-secret
              key: postgresql_postgres_password
        - name: POSTGRES_USER
          value: "sonarUser"
        - name: POSTGRES_PASSWORD
          valueFrom:
            secretKeyRef:
              name: sonarqube-secret
              key: postgresql_password
      volumeMounts:
        - name: secrets
          mountPath: "/mnt/secrets"
          readOnly: true
  volumes:
    - name: secrets
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: sonarqube-secret-provider-class

Log outputs from relevant pods (deployed on same node)..

# secrets-provider-aws-secrets-store-cs
E1207 14:59:29.874846       1 server.go:151] Failure getting secret values from provider type secretsmanager: Failed to fetch secret from all regions: sonarqube

# csi-secrets-store-secrets-store-csi-driver
1207 15:19:40.094039       1 nodeserver.go:241] "failed to mount secrets store object content" err="rpc error: code = Canceled desc = context canceled" pod="sonarqube/nginx-basic-pod"
I1207 15:19:40.094082       1 nodeserver.go:87] "unmounting target path as node publish volume failed" targetPath="/var/lib/kubelet/pods/fe5a2b9c-08e9-4efe-8f9a-14615caf205d/volumes/kubernetes.io~csi/secrets/mount" pod="sonarqube/nginx-basic-pod"

If I comment out the nodeSelector and toleration (free to deploy onto any node) then all works fine and as expected.

Some additional context..

• installed via helm and have syncSecret.enabled=true
• no hyphens in SecretProviderClass, have double quoted all values and specified region
• ServiceAccount, RBAC and IAM Role/permissions all tested and work

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: sonarqube-secret-provider-class
spec:
  provider: aws
  parameters:
    region: eu-west-1
    objects: |
      - objectName: "sonarqube"
        objectType: "secretsmanager"
        jmesPath:
          - path: "login_password"
            objectAlias: "login_password"
          - path: "postgresql_password"
            objectAlias: "postgresql_password"
          - path: "postgresql_postgres_password"
            objectAlias: "postgresql_postgres_password"
  secretObjects:
    - secretName: sonarqube-secret
      type: Opaque
      data:
        - key: "login_password"
          objectName: "login_password"
        - key: "postgresql_password"
          objectName: "postgresql_password"
        - key: "postgresql_postgres_password"
          objectName: "postgresql_postgres_password"

Anyone have any ideas of what might be causing this and how to fix?

[vajeen]

Can you confirm that csi-secrets-store-provider-aws and secrets-store-csi-driver can tolerate your new node taint (sonarqube: "true")

[jbct]

Were you able to resolve the issue?

[jbct]

Closing, please open a new issue if this is still a problem.