Vault agent can’t authenticate using k8s 1.22.5

[mrjebabli]

Hello,
i want to get my secret from vault, this is the first time to integrate vault (vault v1.10.3 ) with k8s in the same cluster and the same namespace.
I’m following this tuto, to get a secret to the application but im getting always errors
kubectl logs $(kubectl get pod -l app=orgchart -o jsonpath="{.items[0].metadata.name}") --container vault-agent error: container vault-agent is not valid for pod orgchart-798cbc6c76-szd9q

and in the log vault agent injector I’m getting :
[ERROR] handler: http: TLS handshake error from 10.1.0.129:52015: remote error: tls: bad certificate
for the certificate i get it using
kubectl config view --raw --minify --flatten --output 'jsonpath={.clusters[].cluster.certificate-authority-data}' | base64 --decode

any idea how to investigate more or how to solve it .
thanks

[adamko147]

I'm seeing the same issue after updating vault agent injector from helm chart v0.19.0 to v0.20.0, which I think uses now hashicorp/vault v1.10.3 and hashicorp/vault-k8s v0.16.0 images.
Temporarily fixed the issue by rolling back to 0.19.0 helm chart.
Running on AWS EKS 1.22 with external vault server.

[mrjebabli]

@adamko147 thanks adam for this workaround, there is option to change the version without delete and reinstall?

[tvoran]

Hi folks, it sounds like you may have been running into some issues that were addressed in v0.16.1 of the vault-k8s injector: https://github.com/hashicorp/vault-k8s/blob/main/CHANGELOG.md#0161-may-25-2022

The v0.20.1 release of the chart includes that as the default vault-k8s version: https://github.com/hashicorp/vault-helm/blob/main/CHANGELOG.md#0201-may-25th-2022

@mrjebabli You should be able to use the helm upgrade command to update the chart version and values: https://helm.sh/docs/helm/helm_upgrade/

[mrjebabli]

@tvoran thanks, the v0.20.1 solves my issue

[dcshiman]

I got the follow error after upgrading to k8s v1.22

│ Error: error running dry run for a diff: current release manifest contains removed kubernetes api(s) for this kubernetes version and it is therefore unable to build the kubernetes objec
ts for performing the diff. error from kubernetes: [unable to recognize "": no matches for kind "ClusterRoleBinding" in version "rbac.authorization.k8s.io/v1beta1", unable to recognize ""
: no matches for kind "RoleBinding" in version "rbac.authorization.k8s.io/v1beta1", unable to recognize "": no matches for kind "MutatingWebhookConfiguration" in version "admissionregistr
ation.k8s.io/v1beta1"]

Had to rename the manifest to do the upgrade. Here is my guide on how to do the fix

[tvoran]

@dcshiman That's good to know! Which versions of kubernetes and the vault helm chart were you on before the upgrade?

[adamko147]

I can confirm that after upgrade to chart https://helm.releases.hashicorp.com/vault@0.20.1 the issue is fixed