Retrieving Secrets from Vault throws 404 without any error text.

[ib-ak]

Retrieving Secrets from Vault throws 404 without any error text.

"payload":{"response":{"statusCode":404,"body":{"errors":[]}}},"msg":"failure while polling the secret <ns>/<external-secret>

I am unable to debug what I am doing wrong.

edit:

more logs

{"level":30,"time":1613896610435,"pid":17,"hostname":"external-secrets-kubernetes-external-secrets-5f45dd675d-lz9jf","msg":"running poll on the secret it-automation/gh-secret"}
{"level":20,"time":1613896610447,"pid":17,"hostname":"external-secrets-kubernetes-external-secrets-5f45dd675d-lz9jf","msg":"fetching new token from vault for role it-automation-runner-role on it/kubernetes"}
{"level":20,"time":1613896610450,"pid":17,"hostname":"external-secrets-kubernetes-external-secrets-5f45dd675d-lz9jf","msg":"fetching new token from vault for role it-automation-runner-role on it/kubernetes"}
{"level":20,"time":1613896610450,"pid":17,"hostname":"external-secrets-kubernetes-external-secrets-5f45dd675d-lz9jf","msg":"fetching new token from vault for role it-automation-runner-role on it/kubernetes"}
{"level":50,"time":1613896610538,"pid":17,"hostname":"external-secrets-kubernetes-external-secrets-5f45dd675d-lz9jf","payload":{"response":{"statusCode":404,"body":{"errors":[]}}},"msg":"failure while polling the secret it-automation/gh-secret"}

[sklarsa]

I'm experiencing the same behavior, but this is when trying to sync kv v2 secrets. For context, I'm using external-secrets v6.1.0 and vault 1.6.2 with the following yaml:

apiVersion: 'kubernetes-client.io/v1'
kind: ExternalSecret
metadata:
  name: hello-vault-service
spec:
  backendType: vault
  vaultMountPoint: kubernetes
  vaultRole: demo
  data:
  - name: passwordv2
    key: secret/data/hello-service/credentials
    property: password
  - name: api-keyv2
    key: secret/data/hello-service/credentials
    property: api-key

And vault kv get secret/data/hello-service/credentials works when running on a separate pod with the same serviceaccount

[Flydiverny]

"statusCode":404 Seems like it can't find the given secret? 😄 So I'm not really sure what other errors it would say, I take it the request succeeded but either credentials are lacking or path is off? 🤔

[sklarsa]

Sorry @Flydiverny, still adding context, but vault kv get secret/data/hello-service/credentials works when running on a separate pod with the same serviceaccount

[Flydiverny]

Maybe its failing to get the token, judging by that being the last message before the errors. In that case maybe the vault mount point? But I honestly don't know to much about vault regardless 😄

If it got past the auth it should trigger this log this._logger.debug(reading secret key ${key} from vault), so indeed seems like its failing when calling kubernetesLogin which is using our vault dependency

[sklarsa]

No prob! I'm still doing what I can to debug. The auth seems to be working with kv v1 secrets, I'm just running into a different error (wanted to keep this ticket clean).
(With v1 secrets and same yaml)

Failed to JSON.parse value for 'kv/data/hello-service/credentials', please verify that your secret value is correctly formatted as JSON. To use plain text secret remove the 'property: password'

So if I'm getting to this point, I'm assuming the auth is working

[Flydiverny]

Seems reasonable.
I know the dependency isn't really perfect for kv v2, but to my understanding it should be possible to get it working. But maybe there's something missing. There are some long standing issues nodevault/node-vault#82 perhaps we need to take a look at what they did in vscode-vault and adapt a similar workaround.

[sklarsa]

Sounds good... I can try to spin up a dev env at some point today to dig into this a little more

[Flydiverny]

Here's an untested idea if you want to experiment :) #642 don't have a vault environment setup currently

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[github-actions]

This issue was closed because it has been stalled for 30 days with no activity.