Sidecar injection on K8s "connection refused" backoff

[kenotsolutions]

Hi,

I just created a rancher test env to test sidecar vault-k8s but after patching the deployment's logs show somehow the created sidecar pod cannot reach the vault server which installed in the same namespace from helm. Please check the installation.

RKE
rke version v1.2.3

kubectl version
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.1", GitCommit:"c4d752765b3bbac2237bf87cf0b1c2e307844666", GitTreeState:"clean", BuildDate:"2020-12-19T07:38:38Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.4", GitCommit:"d360454c9bcd1634cf4cc52d1867af5491dc9c5f", GitTreeState:"clean", BuildDate:"2020-11-11T13:09:17Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}

I followed two articles about it and two of them failed.

https://www.hashicorp.com/blog/injecting-vault-secrets-into-kubernetes-pods-via-a-sidecar

and

https://learn.hashicorp.com/tutorials/vault/kubernetes-sidecar?in=vault/kubernetes

I can see the created the Kubernetes authentication, secret and policy on vault (RKE cluster). I created the serviceaccount and I can see the secrets etc.

kubens demo
kubectl get secrets
app-token-52cn6 kubernetes.io/service-account-token 3 10m

kubectl get serviceaccounts
NAME                   SECRETS   AGE
app                    1         11m

Expected behavior
As the articles show I should see the secret but I got the below error.

2021-01-17T01:56:07.881Z [ERROR] auth.handler: error authenticating: error="Put "http://vault.demo.svc:8200/v1/auth/kubernetes/login": dial tcp 10.43.221.161:8200: connect: connection refused" backoff=2.489499895
2021-01-17T01:56:10.371Z [INFO]  auth.handler: authenticating
2021-01-17T01:56:10.408Z [ERROR] auth.handler: error authenticating: error="Put "http://vault.demo.svc:8200/v1/auth/kubernetes/login": dial tcp 10.43.221.161:8200: connect: connection refused" backoff=1.695983266
Error from server (BadRequest): container "app" in pod "app-5d7d9d556c-5k9b7" is waiting to start: PodInitializing

dial tcp 10.43.221.161:8200: connect: connection refused"

Environment:

• Vault Server Version (retrieve with vault status):
• Vault CLI Version (retrieve with vault version):
• Server Operating System/Architecture:

Vault dev mode

K8s

vault               	demo                   	1       	2021-01-17 01:23:56.965486 +0000 UTC   	deployed	vault-0.9.0                                                                       	1.6.1

kubectl exec -ti vault-0 /bin/sh
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
/ $
/ $
/ $
/ $ cat <<EOF > /home/vault/app-policy.hcl
> path "secret*" {
>   capabilities = ["read"]
> }
> EOF
/ $
/ $ vault policy write app /home/vault/app-policy.hcl
Success! Uploaded policy: app
/ $
/ $
/ $
/ $ vault auth enable kubernetes
Success! Enabled kubernetes auth method at: kubernetes/
/ $
/ $
/ $ vault write auth/kubernetes/config \
>    token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
>    kubernetes_host=https://${KUBERNETES_PORT_443_TCP_ADDR}:443 \
>    kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
Success! Data written to: auth/kubernetes/config
/ $
/ $
/ $ vault write auth/kubernetes/role/myapp \
>    bound_service_account_names=app \
>    bound_service_account_namespaces=demo \
>    policies=app \
>    ttl=1h
Success! Data written to: auth/kubernetes/role/myapp
/ $
/ $
/ $ vault kv put secret/helloworld username=foobaruser password=foobarbazpass
Key              Value
---              -----
created_time     2021-01-17T01:30:44.622588733Z
deletion_time    n/a
destroyed        false
version          1
/ $
/ $
/ $ exit

deployment yaml file

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
  labels:
    app: vault-agent-demo
spec:
  selector:
    matchLabels:
      app: vault-agent-demo
  replicas: 1
  template:
    metadata:
      annotations:
      labels:
        app: vault-agent-demo
    spec:
      serviceAccountName: app
      containers:
      - name: app
        image: jweissig/app:0.0.1
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: app
  labels:
    app: vault-agent-demo

before patch

Demo ns all the pods

kubectl get pods
NAME                                    READY   STATUS     RESTARTS   AGE
app-5cbbc55f67-42xkf                    1/1     Running    0          14m
vault-0                                 1/1     Running    0          22m
vault-agent-injector-7f9d58c84c-b9rh6   1/1     Running    0          22m

patch file

spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/agent-inject-secret-helloworld: "secret/helloworld"
        vault.hashicorp.com/role: "myapp"

after patch

NAME                                    READY   STATUS     RESTARTS   AGE
app-5cbbc55f67-42xkf                    1/1     Running    0          18m
app-5d7d9d556c-5k9b7                    0/2     Init:0/1   0          17m
vault-0                                 1/1     Running    0          26m
vault-agent-injector-7f9d58c84c-b9rh6   1/1     Running    0          26m

I got the above error.
Thank You in Advance

[kenotsolutions]

Hi,

This issue also exactly the same on https://learn.hashicorp.com/tutorials/vault/kubernetes-sidecar?in=vault/kubernetes

I followed the steps and Step 7 of 11 I got the same error.

kubectl logs orgchart-7654cd56f9-lsx5g --all-containers

[tvoran]

Hi @kenotsolutions, this is probably due to a bug that was fixed in hashicorp/vault-helm#446. We'll probably be releasing an updated chart in the next couple weeks.

In the meantime, you can try setting VAULT_DEV_LISTEN_ADDRESS to "0.0.0.0:8200" to work around it. For instance, when installing the chart in the learn guide, use this command line:

helm install vault hashicorp/vault --set "server.dev.enabled=true,server.extraEnvironmentVars.VAULT_DEV_LISTEN_ADDRESS=0.0.0.0:8200"

[kenotsolutions]

Hi @kenotsolutions, this is probably due to a bug that was fixed in hashicorp/vault-helm#446. We'll probably be releasing an updated chart in the next couple weeks.

In the meantime, you can try setting VAULT_DEV_LISTEN_ADDRESS to "0.0.0.0:8200" to work around it. For instance, when installing the chart in the learn guide, use this command line:

helm install vault hashicorp/vault --set "server.dev.enabled=true,server.extraEnvironmentVars.VAULT_DEV_LISTEN_ADDRESS=0.0.0.0:8200"

Thanks for letting me know that. I will try your workaround.

[tvoran]

The bug fix hashicorp/vault-helm#446 was included in vault-helm 0.9.1, so closing this for now.