grpc.code=Unknown err="error making mount request: failed to login: 403 errors

[tirvan]

Hi,

I am using Azure managed kubernetes with version 1.21.2, followed the instruction given here step by steps.
https://www.hashicorp.com/blog/retrieve-hashicorp-vault-secrets-with-kubernetes-csi

I double check many times the steps but don't seems to miss anything. I have enabled debug mode for csi and here are the errors.

Service account "internal-app" is already created. The error might seems obvious but I totally no clue what did I miss, been googling for few days but could not find the solution to my problem.

Appreciate your help.

2021-09-16T05:25:22.406Z [INFO]  server: Processing unary gRPC call: grpc.method=/v1alpha1.CSIDriverProvider/Mount
2021-09-16T05:25:22.406Z [DEBUG] server: Request contents: req="attributes:"{\"csi.storage.k8s.io/pod.name\":\"alpine-6c9bf95845-xnmg5\",\"csi.storage.k8s.io/pod.namespace\":\"development\",\"csi.storage.k8s.io/pod.uid\":\"f40d2c4a-0ff9-4459-bad1-70c88a9811db\",\"csi.storage.k8s.io/serviceAccount.name\":\"internal-app\",\"csi.storage.k8s.io/serviceAccount.tokens\":\"\",\"objects\":\"- objectName: dbUsername\\n  secretPath: internal/data/database/config\\n  secretKey: username\\n- objectName: dbPassword\\n  secretPath: internal/data/database/config\\n  secretKey: password\\n\",\"roleName\":\"internal-app\",\"vaultAddress\":\"http://atwin-hashicorp-vault.development.svc.cluster.local:8200\"}" secrets:"null" target_path:"/var/lib/kubelet/pods/f40d2c4a-0ff9-4459-bad1-70c88a9811db/volumes/kubernetes.io~csi/vault-db-creds/mount" permission:"420""
2021-09-16T05:25:22.407Z [DEBUG] server.provider: performing vault login
2021-09-16T05:25:22.407Z [DEBUG] server.provider: creating service account token bound to pod: namespace=development serviceAccountName=internal-app podName=alpine-6c9bf95845-xnmg5 podUID=f40d2c4a-0ff9-4459-bad1-70c88a9811db
2021-09-16T05:25:22.451Z [DEBUG] server.provider: service account token creation successful
2021-09-16T05:25:22.528Z [INFO]  server: Finished unary gRPC call: grpc.method=/v1alpha1.CSIDriverProvider/Mount grpc.time=121.554688ms grpc.code=Unknown err="error making mount request: failed to login: Error making API request.

URL: POST http://hashicorp-vault.development.svc.cluster.local:8200/v1/auth/kubernetes/login
Code: 403. Errors:

* permission denied"

My secret store

---
apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
  name: vault-database
  namespace: development
spec:
  provider: vault
  secretObjects:
    - secretName: vault-db-secrets
      type: Opaque
      data:
        - objectName: dbUsername # References dbUsername below
          key: username # Key within k8s secret for this value
        - objectName: dbPassword
          key: password
  parameters:
    roleName: internal-app
    vaultAddress: "http://hashicorp-vault.development.svc.cluster.local:8200"
    # vaultNamespace: "development"
    # vaultCACertPath: "/mnt/tls/ca.crt"
    # vaultTLSClientCertPath: "/mnt/tls/tls.crt"
    # vaultTLSClientKeyPath: "/mnt/tls/tls.key"
    objects: |
      - objectName: dbUsername
        secretPath: internal/data/database/config
        secretKey: username
      - objectName: dbPassword
        secretPath: internal/data/database/config
        secretKey: password

apiVersion: apps/v1
kind: Deployment
metadata:
  name: alpine
  namespace: development
  labels:
    app: alpine
spec:
  replicas: 1
  selector:
    matchLabels:
      app: alpine
  template:
    metadata:
      annotations:
      labels:
        app: alpine
    spec:
      serviceAccountName: internal-app

      containers:
        - name: alpine
          image: alpine:latest
          command: ["/bin/sh", "-c"]
          args: 
            - sleep 600
          env:
            - name: DB_USERNAME
              valueFrom:
                secretKeyRef:
                  name: vault-db-creds-secrets
                  key: username
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: vault-db-creds-secrets
                  key: password
          volumeMounts:
            - name: vault-db-creds
              mountPath: '/mnt/secrets-store'
              readOnly: true

      volumes:
        - name: vault-db-creds
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: vault-database

[adrafiq]

Can you paste your audit logs from vault server. Same timeline?

[tirvan]

Hi @adrafiq ,

The error I got from vault server is the following. I am only testing with one pod and there are many errors repeating like below.

2021-09-19T13:01:37.573Z [ERROR] auth.kubernetes.auth_kubernetes_846d0ba1: login unauthorized due to: lookup failed: service account unauthorized; this could mean it has been deleted or recreated with a new token

I am not sure what I missed, I am still pretty new in kubernetes. Appreciate your help a lot.

#### enable auth kubernetes
vault auth enable kubernetes
vault secrets enable -path=internal kv-v2
vault kv put internal/database/config username="db-readonly-username" password="db-secret-password"
vault kv get internal/database/config

#### Get the issuer
curl --silent http://127.0.0.1:8001/.well-known/openid-configuration | jq -r .issuer

#### Writing to Kubernetes configuration

vault write auth/kubernetes/config \
  issuer="https://<using above output>" \
  token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
  kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \
  kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

vault policy write internal-app - <<EOH
path "internal/data/database/config" {
  capabilities = ["read"]
}
EOH

#### Enable 'internal-app' roles

vault write auth/kubernetes/role/internal-app \
  bound_service_account_names=internal-app \
  bound_service_account_namespaces=development \
  policies=internal-app \
  ttl=24h

[tvoran]

Hi @tirvan, by chance was your vault pod deleted and recreated at some point? With K8s 1.21, the vault pod will probably be using an ephemeral projected service token, which only lasts the lifetime of the pod. So that might be why the vault error indicates that the token in the k8s auth config is no longer valid. You could try re-applying that config to see if that helps.

Otherwise you can try using the service account's default token when setting up the k8s auth method. Its issuer will be the default of kubernetes/serviceaccount. It'll be present in a secret in the vault pod's namespace, named something like hashicorp-vault-token-xxxxx (if I'm reading your deployment correctly):

VAULT_SECRET_NAME=$(kubectl get secrets -n development --output=json | jq -r '.items[].metadata | select(.name|startswith("hashicorp-vault-token-")).name')
kubectl get secret -n development $VAULT_SECRET_NAME --output='go-template={{ .data.token }}' | base64 --decode

[tirvan]

Hi @tvoran ,

Thank you very much. You are right about vault pod got recreated at some point, after checking the age, the vault pod is indeed got killed/restarted as the csi provider is older in age.

After re-applying the config, everything works great and I am able to see my secrets now.

Again, appreciate so much for your help.

Regards,
Irvan

[vagharsh]

hello, i am getting the same 403 error .. here is my audit log

{
        "time": "2024-05-21T07:56:17.187539563Z",
        "type": "request",
        "auth": {
            "policy_results": {
                "allowed": true
            },
            "token_type": "default"
        },
        "request": {
            "id": "b4b458b4-f452-ea03-b89b-838e19a2ed0c",
            "operation": "update",
            "mount_point": "auth/test/",
            "mount_type": "kubernetes",
            "mount_accessor": "auth_kubernetes_4f1c1b0f",
            "mount_running_version": "v0.17.1+builtin",
            "mount_class": "auth",
            "namespace": {
                "id": "root"
            },
            "path": "auth/test/login",
            "data": {
                "jwt": "hmac-sha256:6f98da034f83d374998df47e4d227d4706dc01495fdf09cf99750569c3a678bc",
                "role": "hmac-sha256:ff48cb037f6b4b07e1b84b77fd97b65e29571da672a90f2c88d0cc917f04867d"
            },
            "remote_address": "10.77.43.15",
            "remote_port": 52073
        }
    }

and from my csi driver pods i am getting the following error

2024-05-21T07:58:05.238Z [INFO]  server: Processing unary gRPC call: grpc.method=/v1alpha1.CSIDriverProvider/Mount
2024-05-21T07:58:35.277Z [INFO]  server: Finished unary gRPC call: grpc.method=/v1alpha1.CSIDriverProvider/Mount grpc.time=30.038993184s grpc.code=Unknown
  err=
  | error making mount request: couldn't read secret "general-port": failed to login: Error making API request.
  |
  | URL: POST http://vault-internal.vault.svc.cluster.local:8200/v1/auth/test/login
  | Code: 403. Errors:
  |
  | * permission denied

FYI : my vault cluster is in another EKS cluster i have followed the documentation here https://developer.hashicorp.com/vault/tutorials/kubernetes/kubernetes-external-vault#configure-kubernetes-authentication