TLS error in basic secret injection video

[wonboyn]

Hey Marcel,

I was trying to replicate the setup from the video "Basic secret injection for microservices on Kubernetes using Vault".
I got to the point of starting the example app deployment & found that the pod starts but stays in the "Init:0/1" status.

The vault injector pod logs show that it received the mutating webhook call:

kubectl -n vault-example logs vault-example-agent-injector-7cdd648787-tv4lb
2020-08-12T22:55:14.523Z [INFO] handler: Starting handler..
Listening on ":8080"...
Updated certificate bundle received. Updating certs...
2020-08-12T23:08:00.894Z [INFO] handler: Request received: Method=POST URL=/mutate?timeout=30s

The logs from the vault pod show a TLS error:

kubectl -n vault-example logs vault-example-0
==> Vault server configuration:

         Api Address: https://10.244.0.6:8200
                 Cgo: disabled
     Cluster Address: https://10.244.0.6:8201
          Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "[::]:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "enabled")
           Log Level: info
               Mlock: supported: true, enabled: false
       Recovery Mode: false
             Storage: file
             Version: Vault v1.3.1

2020-08-12T22:50:10.226Z [INFO] proxy environment: http_proxy= https_proxy= no_proxy=
==> Vault server started! Log data will stream in below:

2020-08-12T22:50:50.416Z [INFO] core.cluster-listener: starting listener: listener_address=[::]:8201
2020-08-12T22:50:50.416Z [INFO] core.cluster-listener: serving cluster requests: cluster_listen_address=[::]:8201
2020-08-12T22:50:50.416Z [INFO] core: post-unseal setup starting
2020-08-12T22:50:50.417Z [INFO] core: loaded wrapping token key
2020-08-12T22:50:50.417Z [INFO] core: successfully setup plugin catalog: plugin-directory=
2020-08-12T22:50:50.418Z [INFO] core: successfully mounted backend: type=system path=sys/
2020-08-12T22:50:50.418Z [INFO] core: successfully mounted backend: type=identity path=identity/
2020-08-12T22:50:50.419Z [INFO] core: successfully mounted backend: type=cubbyhole path=cubbyhole/
2020-08-12T22:50:50.421Z [INFO] core: successfully enabled credential backend: type=token path=token/
2020-08-12T22:50:50.421Z [INFO] core: restoring leases
2020-08-12T22:50:50.421Z [INFO] rollback: starting rollback manager
2020-08-12T22:50:50.422Z [INFO] identity: entities restored
2020-08-12T22:50:50.422Z [INFO] expiration: lease restore complete
2020-08-12T22:50:50.422Z [INFO] identity: groups restored
2020-08-12T22:50:50.422Z [INFO] core: post-unseal setup complete
2020-08-12T22:50:50.423Z [INFO] core: vault is unsealed
2020-08-12T23:01:10.547Z [INFO] core: enabled credential backend: path=kubernetes/ type=kubernetes
2020-08-12T23:05:51.876Z [INFO] core: successful mount: namespace= path=secret/ type=kv
2020-08-12T23:06:38.902Z [INFO] http: TLS handshake error from 127.0.0.1:52998: remote error: tls: unknown certificate

And the logs from the init container show an error trying to authenticate with vault:

kubectl -n vault-example logs basic-secret-74b4fdbcdc-2zmtl -c vault-agent-init
==> Vault server started! Log data will stream in below:

==> Vault agent configuration:
2020-08-12T23:08:01.568Z [INFO] sink.file: creating file sink
2020-08-12T23:08:01.568Z [INFO] sink.file: file sink configured: path=/home/vault/.token mode=-rw-r-----
2020-08-12T23:08:01.568Z [INFO] auth.handler: starting auth handler
2020-08-12T23:08:01.568Z [INFO] auth.handler: authenticating
2020-08-12T23:08:01.568Z [INFO] sink.server: starting sink server

2020-08-12T23:08:01.568Z [INFO] template.server: starting template server
Cgo: disabled
Log Level: info
Version: Vault v1.3.1

2020/08/12 23:08:01.569034 [INFO] (runner) creating new runner (dry: false, once: false)
2020/08/12 23:08:01.569618 [WARN] (clients) disabling vault SSL verification
2020/08/12 23:08:01.569658 [INFO] (runner) creating watcher
2020-08-12T23:08:11.580Z [ERROR] auth.handler: error authenticating: error="Put https://vault-example.vault-example.svc:8200/v1/auth/kubernetes/login: dial tcp: lookup vault-example.vault-example.svc on 10.96.0.10:53: read udp 10.244.0.8:50821->10.96.0.10:53: read: connection refused" backoff=2.156164762
2020-08-12T23:08:13.703Z [INFO] auth.handler: authenticating
2020-08-12T23:08:23.712Z [ERROR] auth.handler: error authenticating: error="Put https://vault-example.vault-example.svc:8200/v1/auth/kubernetes/login: dial tcp: lookup vault-example.vault-example.svc on 10.96.0.10:53: read udp 10.244.0.8:41477->10.96.0.10:53: i/o timeout" backoff=2.29257713

In terms of TLS - I used the exact TLS config/process indicated in your ssl_generate_self_signed.txt file.

Any suggestions would be greatly appreciated.

Thanks

Tim

[marcel-dempers]

Hey Tim,
Based on this error you're getting read error on UDP port (53).
Vault agent is unable to read DNS:

read udp 10.244.0.8:41477->10.96.0.10:53: i/o timeout" backoff=2.29257713

It appears DNS in your cluster is not working correctly

[wonboyn]

Thanks for that - you're right about it being a dns issue.
The two coredns pods have crashed. Now need to figure out why.

[wonboyn]

The issue seems to have been related to the coredns version used in the kind image v1.17.0 (at least when running on macos). When I deleted the kind cluster & recreated it - the coredns pods failed straight away.

I switched to using v1.18.6 and the issue went away and the secret injection worked.

Thanks again