Skip to content
This repository has been archived by the owner on Jul 26, 2022. It is now read-only.

Retrieving Secrets from Vault throws 404 without any error text. #630

Closed
ib-ak opened this issue Feb 21, 2021 · 10 comments
Closed

Retrieving Secrets from Vault throws 404 without any error text. #630

ib-ak opened this issue Feb 21, 2021 · 10 comments
Labels

Comments

@ib-ak
Copy link

ib-ak commented Feb 21, 2021

Retrieving Secrets from Vault throws 404 without any error text.

"payload":{"response":{"statusCode":404,"body":{"errors":[]}}},"msg":"failure while polling the secret <ns>/<external-secret>

I am unable to debug what I am doing wrong.

edit:

more logs

{"level":30,"time":1613896610435,"pid":17,"hostname":"external-secrets-kubernetes-external-secrets-5f45dd675d-lz9jf","msg":"running poll on the secret it-automation/gh-secret"}
{"level":20,"time":1613896610447,"pid":17,"hostname":"external-secrets-kubernetes-external-secrets-5f45dd675d-lz9jf","msg":"fetching new token from vault for role it-automation-runner-role on it/kubernetes"}
{"level":20,"time":1613896610450,"pid":17,"hostname":"external-secrets-kubernetes-external-secrets-5f45dd675d-lz9jf","msg":"fetching new token from vault for role it-automation-runner-role on it/kubernetes"}
{"level":20,"time":1613896610450,"pid":17,"hostname":"external-secrets-kubernetes-external-secrets-5f45dd675d-lz9jf","msg":"fetching new token from vault for role it-automation-runner-role on it/kubernetes"}
{"level":50,"time":1613896610538,"pid":17,"hostname":"external-secrets-kubernetes-external-secrets-5f45dd675d-lz9jf","payload":{"response":{"statusCode":404,"body":{"errors":[]}}},"msg":"failure while polling the secret it-automation/gh-secret"}
@sklarsa
Copy link

sklarsa commented Feb 25, 2021

I'm experiencing the same behavior, but this is when trying to sync kv v2 secrets. For context, I'm using external-secrets v6.1.0 and vault 1.6.2 with the following yaml:

apiVersion: 'kubernetes-client.io/v1'
kind: ExternalSecret
metadata:
  name: hello-vault-service
spec:
  backendType: vault
  vaultMountPoint: kubernetes
  vaultRole: demo
  data:
  - name: passwordv2
    key: secret/data/hello-service/credentials
    property: password
  - name: api-keyv2
    key: secret/data/hello-service/credentials
    property: api-key

And vault kv get secret/data/hello-service/credentials works when running on a separate pod with the same serviceaccount

@Flydiverny
Copy link
Member

Flydiverny commented Feb 25, 2021

"statusCode":404 Seems like it can't find the given secret? 😄 So I'm not really sure what other errors it would say, I take it the request succeeded but either credentials are lacking or path is off? 🤔

@sklarsa
Copy link

sklarsa commented Feb 25, 2021

Sorry @Flydiverny, still adding context, but vault kv get secret/data/hello-service/credentials works when running on a separate pod with the same serviceaccount

@Flydiverny
Copy link
Member

Flydiverny commented Feb 25, 2021

Maybe its failing to get the token, judging by that being the last message before the errors. In that case maybe the vault mount point? But I honestly don't know to much about vault regardless 😄

If it got past the auth it should trigger this log this._logger.debug(reading secret key ${key} from vault), so indeed seems like its failing when calling kubernetesLogin which is using our vault dependency

@sklarsa
Copy link

sklarsa commented Feb 25, 2021

No prob! I'm still doing what I can to debug. The auth seems to be working with kv v1 secrets, I'm just running into a different error (wanted to keep this ticket clean).
(With v1 secrets and same yaml)

Failed to JSON.parse value for 'kv/data/hello-service/credentials', please verify that your secret value is correctly formatted as JSON. To use plain text secret remove the 'property: password'

So if I'm getting to this point, I'm assuming the auth is working

@Flydiverny
Copy link
Member

Seems reasonable.
I know the dependency isn't really perfect for kv v2, but to my understanding it should be possible to get it working. But maybe there's something missing. There are some long standing issues nodevault/node-vault#82 perhaps we need to take a look at what they did in vscode-vault and adapt a similar workaround.

@sklarsa
Copy link

sklarsa commented Feb 25, 2021

Sounds good... I can try to spin up a dev env at some point today to dig into this a little more

@Flydiverny
Copy link
Member

Here's an untested idea if you want to experiment :) #642 don't have a vault environment setup currently

@github-actions
Copy link

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 30 days.

@github-actions github-actions bot added the Stale label May 27, 2021
@github-actions
Copy link

This issue was closed because it has been stalled for 30 days with no activity.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

No branches or pull requests

3 participants