Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Injector with external Vault server? #15

Open
smurfralf opened this issue Dec 20, 2019 · 18 comments
Open

Injector with external Vault server? #15

smurfralf opened this issue Dec 20, 2019 · 18 comments
Labels
Milestone

Comments

@smurfralf
Copy link

@smurfralf smurfralf commented Dec 20, 2019

The scenario we want to support is to use a vault server which pre-exists the kubernetes cluster. We want the vault-k8s injector capability to talk to this vault server.

Is this scenario supported? I see that the injector command (vault-k8s/subcommand/injector/flags.go) takes a vault-address argument. And I see that deployment (vault-helm/templates/injector-deployment.yaml) sets the AGENT_INJECT_VAULT_ADDR env variable.

If the injector can be configured to use an external service, please document an example configuration that also disables installing vault as a service inside kubernetes.

@jasonodonnell

This comment has been minimized.

Copy link
Contributor

@jasonodonnell jasonodonnell commented Dec 20, 2019

@smurfralf This scenario should work, provided pods can communicate with the external Vault cluster. You will need to set AGENT_INJECT_VAULT_ADDR on the deployment, as you already noticed. I will document the manual installation process using the deploy files in this repo.

@ls-brentsmith

This comment has been minimized.

Copy link

@ls-brentsmith ls-brentsmith commented Dec 22, 2019

I'm struggling with using the deploy manifests from this repo, which i'm attempting to use because the helm charts are currently not working as intended with regards to global.enable variable, among other templating issues.

FWIW - I was able to get it working by changing the following

1) I added the following rbac configuration which is reused from the kubernetes auth documentation, but it's not clear to me why, as it seems the admission registration should cover it.
2) I changed AGENT_INJECT_VAULT_ADDR to my vault server url
3) I needed to make sure the vault-injector service account existed in the same namespace as my application, which makes sense in retrospect, but wasn't obvious to me in the manifests or the charts (nothing is ever obvious in a helm chart to be fair :) )

I will report back here as I become less confused :)

@jasonodonnell

This comment has been minimized.

Copy link
Contributor

@jasonodonnell jasonodonnell commented Dec 22, 2019

Hi @ls-brentsmith,

I added the following rbac configuration which is reused from the kubernetes auth documentation, but it's not clear to me why, as it seems the admission registration should cover it.

That RBAC is not needed by the injector. The injector doesn't actually communicate with Vault at all. The rbac you listed is what Vault uses when authenticating service accounts in Kubernetes.

I changed AGENT_INJECT_VAULT_ADDR to my vault server url

This should really be the only thing you need to do.

I needed to make sure the vault-injector service account existed in the same namespace as my application, which makes sense in retrospect, but wasn't obvious to me in the manifests or the charts (nothing is ever obvious in a helm chart to be fair :) )

The deploy examples in this repository use the vault namespace. You should only need to create a namespace vault and run the following (assuming you configured the AGENT_INJECT_VAULT_ADDR with your external Vault server:

kubectl create namespace vault
kubectl create -f ./deploy/injector-rbac.yaml --namespace=vault
kubectl create -f ./deploy/injector-service.yaml --namespace=vault
kubectl create -f ./deploy/injector-mutating-webhook.yaml --namespace=vault
kubectl create -f ./deploy/deployment.yaml --namespace=vault
kubectl get pods --namespace=vault
@jasonodonnell jasonodonnell added the docs label Dec 22, 2019
@ls-brentsmith

This comment has been minimized.

Copy link

@ls-brentsmith ls-brentsmith commented Dec 22, 2019

@jasonodonnell thanks for the quick response.
In the light of morning I saw why I stumbled there.

Follow up question, is it possible to inject a custom auth path for the vault-agents?

@jasonodonnell

This comment has been minimized.

Copy link
Contributor

@jasonodonnell jasonodonnell commented Dec 23, 2019

@ls-brentsmith, there's no annotation currently that lets you alter the auth path, however, you can mount your own Vault Agent configuration files using a configmap. Here's an example: https://www.vaultproject.io/docs/platform/k8s/injector/examples.html#configmap-example. Mount path would go in the method section: https://www.vaultproject.io/docs/agent/autoauth/index.html#mount_path

Hope that helps!

@ls-brentsmith

This comment has been minimized.

Copy link

@ls-brentsmith ls-brentsmith commented Dec 23, 2019

Thank you for the quick response!
I was just about to come comment the same thing.

@buckner

This comment has been minimized.

Copy link

@buckner buckner commented Dec 23, 2019

Are there plans to add an annotation for custom auth paths? It'd be nice to have a choice between a configmap or just annotations.

@jasonodonnell

This comment has been minimized.

Copy link
Contributor

@jasonodonnell jasonodonnell commented Dec 23, 2019

Yes, I'll be adding one @buckner!

@smurfralf

This comment has been minimized.

Copy link
Author

@smurfralf smurfralf commented Dec 23, 2019

@jasonodonnell - back to the original topic of this issue...
Your comment WRT setting external URL for AGENT_INJECT_VAULT_ADDR was: "This should really be the only thing you need to do." I did it using name "vault-inj" for the helm install, and the injector is indeed functional. Yay!
But my k8s cluster also has a zombie pod vault-inj-0 showing
Readiness probe failed
because of course this vault server hasn't been initialized.
When I made a stab at setting helm value service.enabled: false then that breaks the injector where logs show
2019-12-23T16:40:45.791Z [INFO] handler: Starting handler.. Listening on ":8080"... Updated certificate bundle received. Updating certs... 2019/12/23 16:40:46 http: TLS handshake error from 10.0.16.98:41948: no certificate available
My guess is that you have dependencies there in your automagic configuration of TLS?

At any rate, solving that isn't my point. What I asked for in the description would be nice:

... please document an example configuration that (allows configuration of an external vault server [preferably without changing the vault-helm template] and that) also disables installing vault as a service inside kubernetes.

@jasonodonnell

This comment has been minimized.

Copy link
Contributor

@jasonodonnell jasonodonnell commented Dec 23, 2019

@smurfralf There are some dependencies in the Vault Helm project between the projects, so some stuff will need to change over there for installing the injector standalone.

I'll document the process of configuring the injector with an external Vault service after the new year since HashiCorp is currently in a holiday shutdown.

@VinothChinnadurai

This comment has been minimized.

Copy link

@VinothChinnadurai VinothChinnadurai commented Dec 24, 2019

@jasonodonnell I have a setup of external vault server with consul backend and placed a load balancer in front of vault server in an EKS environment. Now m app has been deployed in separate EKS cluster where I mentioned AGENT_INJECT_VAULT_ADDR as the above mentioned ELB(Port 80). Now my app pod becomes not ready after I injected the init container . STATUS becomes 'Init:0/1' . I think it is something related to token initialization. How can I mention that token here and retrieve the secrets inside the Pod?

@jasonodonnell

This comment has been minimized.

Copy link
Contributor

@jasonodonnell jasonodonnell commented Dec 24, 2019

@VinothChinnadurai Did you check the init container logs to see what's happening?

kubectl logs <name of pod> -c vault-agent-init

@VinothChinnadurai

This comment has been minimized.

Copy link

@VinothChinnadurai VinothChinnadurai commented Dec 24, 2019

Yes @jasonodonnell . Please find it here
`URL: PUT http://vault.demo.svc:8200/v1/auth/kubernetes/login
Code: 400. Errors:

  • invalid role name "exampleapp-role"" backoff=1.580907647
    2019-12-24T10:09:31.660Z [INFO] auth.handler: authenticating
    2019-12-24T10:09:31.667Z [ERROR] auth.handler: error authenticating: error="Error making API request`

Not sure why it still hits some other service(vault.demo.svc) instead of ELB address

shlee322 added a commit to shlee322/vault-k8s that referenced this issue Dec 24, 2019
@jasonodonnell

This comment has been minimized.

Copy link
Contributor

@jasonodonnell jasonodonnell commented Dec 24, 2019

@VinothChinnadurai Double check the env is set correctly because that looks like the default

value: "https://vault.$(NAMESPACE).svc:8200"

Next retrigger the injection by patching the pod. You'll need to set the status annotation to vault.hashicorp.com/agent-inject-status: update to retrigger the injection (the config needs to be refreshed on the agent).

@VinothChinnadurai

This comment has been minimized.

Copy link

@VinothChinnadurai VinothChinnadurai commented Jan 2, 2020

@jasonodonnell Happy New year! Thanks. It worked for me! But I find an issue with the syncing part where I have updated my secrets through vault API command
curl \ --header "X-Vault-Token: xxx" \ --request POST \ --data @payload.json \ http://myelbaddress:8200/v1/secret/helloworld

Payload.json has
{ "data": { "foo": "bar", "zip": "zap" } }

Post then in the GET command of API, I can receive the updated secrets.
I tried to see the same by logging inside my app pod and verified the file /vault/secret/helloworld
Where it still showed the older values.
Kindly help me with what might be the cause.

@jasonodonnell jasonodonnell added this to the 0.2.0 milestone Jan 2, 2020
@jasonodonnell

This comment has been minimized.

Copy link
Contributor

@jasonodonnell jasonodonnell commented Jan 2, 2020

@VinothChinnadurai Lower the TTL of your secret. Updates are based on TTL of the secret.

@trx35479

This comment has been minimized.

Copy link

@trx35479 trx35479 commented Jan 14, 2020

@jasonodonnell does this also work with istio service mesh?

@malnick

This comment has been minimized.

Copy link

@malnick malnick commented Jan 23, 2020

@trx35479 We haven't done explicit testing with istio yet, so we can not verify if this will or will not work with it. If you happen to test this, please let us know what you find.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
8 participants
You can’t perform that action at this time.