Injector with External vault service

[whume]

Hello I am testing the integration with and found #15 which helped with a few things but I hit a wall and am struggling to find where to go from here do to lack of information.

So I updated the Vault URL in the injector-deployment.yaml file as suggested and I see my injector attempting to connect. After deploying all the files. I am also updating the auth path as mine is custom.

This is all deployed in namespace called vault

I the integrate with the vault cluster using the following.

k8s_host="$(kubectl config view --minify | grep server | cut -f 2- -d ":" | tr -d " ")"
k8s_cacert="$(kubectl config view --raw --minify --flatten -o jsonpath='{.clusters[].cluster.certificate-authority-data}' | base64 --decode)"
secret_name="$(kubectl get serviceaccount vault-injector -n vault -o go-template='{{ (index .secrets 0).name }}')"
tr_account_token="$(kubectl get secret ${secret_name} -n vault -o go-template='{{ .data.token }}' | base64 --decode)"

Then add to vault
vault write auth/cluster-01/config token_reviewer_jwt="${tr_account_token}" kubernetes_host="${k8s_host}" kubernetes_ca_cert="${k8s_cacert}"

Create a vault role:
vault write auth/cluster-01/role/myapp bound_service_account_names=app bound_service_account_namespaces="*" policies=app ttl=1h
Below is the app config I am using in the test namespace

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
  labels:
    app: vault-agent-demo
spec:
  selector:
    matchLabels:
      app: vault-agent-demo
  replicas: 1
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/agent-inject-secret-helloworld: "secret/helloworld"
        vault.hashicorp.com/role: "myapp"
        vault.hashicorp.com/auth-path: "/auth/cluster-01"
      labels:
        app: vault-agent-demo
    spec:
      serviceAccountName: app
      containers:
        - name: app
          image: nginxdemos/hello
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: app
  labels:
    app: vault-agent-demo

Then I get these logs in the pod

2020-02-04T18:20:32.811Z [ERROR] auth.handler: error authenticating: error="Error making API request.
URL: PUT https://uri.company.com/v1/auth/cluster-01/login
Code: 403. Errors:

• permission denied" backoff=2.0216039

Any ideas?

Curl returns the same response :(

[jasonodonnell]

Hi @whume, I believe you're using the wrong service account. Vault requires RBAC to communicate with Kubernetes to verify service accounts. Looks like you're using the injector service account which is not the same thing.

Create the service account:

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: vault-auth
  namespace: vault
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: role-tokenreview-binding
  namespace: vault
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
  - kind: ServiceAccount
    name: vault-auth
    namespace: vault

Get the JWT for that service account:

secret_name="$(kubectl get serviceaccount vault-auth -n vault -o go-template='{{ (index .secrets 0).name }}')"
tr_account_token="$(kubectl get secret ${secret_name} -n vault -o go-template='{{ .data.token }}' | base64 --decode)"

Setup the Vault auth method using that new JWT:

vault write auth/cluster-01/config token_reviewer_jwt="${tr_account_token}" kubernetes_host="${k8s_host}" kubernetes_ca_cert="${k8s_cacert}"

[whume]

Sadly I am still getting the same thing after applying the above. Error has not changed just permission denied

[whume]

So we figured it out. Turns out everything in the cluster was good Vault was being blocked by the firewall trying to send back to the cluster. We are good as far as the issue. If you don't mind I have one more question. How is the TTL for the rotation on the secret handled. Is that controlled by the TTL vault role or is there a setting to adjust that?

Thanks for the help.

[jasonodonnell]

Hi @whume ,

Token rotations are based on the token_max_ttl for the role:

vault write auth/kubernetes/role/db-app \
    bound_service_account_names=app \
    bound_service_account_namespaces=app \
    policies=app \
    token_max_ttl=60s \
    ttl=30s

Secret rotations are based on the secrets TTL.

Hope that helps! If you're satisfied with these answers, please close the issue!

[whume]

Perfect Thank you

[pksurferdad]

Hi @whume I know this issue is closed but do you remember what the firewall issue was here #69 (comment)? Did you have top allow 8080 TCP traffic? I'm running vault on a stand alone AWS eks k8s cluster and I'm trying to connect to the vault server from another eks cluster .

[chinmaychandak]

@whume, @pksurferdad I'm facing the exact same issue - how were you guys able to solve the firewall issue? I'm using the Vault Agent Injector on an AWS EKS standalone K8s cluster against an external Vault.

[pksurferdad]

sorry @chinmaychandak, is issue turned out not to be firewall related, but once i had the kubernetes auth config correct in vault, my connections were fine; however, i'm running everything on k8s, but needed a k8s cluster to talk to vault on another k8s cluster.

[chinmaychandak]

@pksurferdad, thanks for responding! Yes, I got my setup working too - same issue where I had to reconfigure K8s auth in Vault

[asl-cloud99]

@chinmaychandak @pksurferdad - How did you guys resolved the issue. My am facing the same issue with 2 private GKE cluster.
Error in Vault cluster logs -
[ERROR] auth.kubernetes.auth_kubernetes_08c36485: login unauthorized due to: Post "https://<app_ip_cluster>/apis/authentication.k8s.io/v1/tokenreviews": dial tcp <app_ip_cluster:443: i/o timeout

and application cluster
"errors": [ 46 "permission denied"
I am using following link - (external vault ) https://learn.hashicorp.com/tutorials/vault/agent-kubernetes

[chinmaychandak]

@asl-cloud99, I was able to fix my issues by ensuring 2 things:

• That the K8s Client Cluster's auth config was correctly set up in Vault (try reconfiguring this from scratch; in my case, the K8s JWT_TOKEN had changed because of some changes, and so reconfiguring the auth in Vault helped)
• See if your Client and Vault can talk to each other, i.e. I modified their network inbound rules and such on AWS. You might need to do the same on GCP.