vault-injector-certs Secret metadata is regularly overwritten, breaking any gitops flow

[r0bj]

Describe the bug
During Agent Sidecar Injector installation via helm, Secret vault-injector-certs is created empty:

apiVersion: v1
kind: Secret
metadata:
  name: vault-injector-certs
  namespace: vault-k8s
  labels:
    app.kubernetes.io/name: vault-agent-injector
    app.kubernetes.io/instance: vault
    app.kubernetes.io/managed-by: Helm

Then, Secret is filled with data (certificate) probably by some controller but unfortunately this process is removing metadata (labels and annotations) from the Secret:

apiVersion: v1
kind: Secret
metadata:
  name: vault-injector-certs
  namespace: vault-k8s
data:
  cert: <redacted>
  key: <redacted>
type: Opaque

So any gitops based flow (e.g. with ArgoCD) is broken due to reconciliation loop constantly detecting changes that need to be sync with the source of true (helm).
Additionally by removing annotations and labels all object tracking info is wiped out from the object (ArgoCD is using labels or annotation to keep track of managed objects)

To Reproduce
Steps to reproduce the behavior:

1. Deploy vault-agent injection via helm
2. Wait until Secret vault-injector-certs is filled with data
3. Secret metadata (labels and annotations) is removed.

Expected behavior
Controller or other mechanism is not removing metadata from Secret vault-agent injection during filling it with data.

Environment

• Kubernetes version:

Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.3", GitCommit:"aef86a93758dc3cb2c658dd9657ab4ad4afc21cb", GitTreeState:"clean", BuildDate:"2022-07-13T14:21:56Z", GoVersion:"go1.18.4", Compiler:"gc", Platform:"darwin/arm64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.3", GitCommit:"aef86a93758dc3cb2c658dd9657ab4ad4afc21cb", GitTreeState:"clean", BuildDate:"2022-07-13T14:23:26Z", GoVersion:"go1.18.3", Compiler:"gc", Platform:"linux/amd64"}

• vault-k8s version: 0.16.1

[tvoran]

Hi @r0bj, thanks for raising this issue. I'm seeing the same thing happening with the current version. vault-k8s is populating that Secret with the cert material it generates for communication between it and the kubernetes API.

It looks like we'll need to do a .Get() before the .Update here, or perhaps use `.Patch() instead.

Using something like cert-manager to manage the certs may be a workaround until this is fixed.

[sidewinder12s]

Ya, with ArgoCD I was able to just ignore the labels field as well.