[Bug] Sync Secrets failing to keep secrets in sync

[eitah]

Kyverno Version

1.12

Kubernetes Version

1.28

Kubernetes Platform

GKE

Description

When I raise sync secret policy and i update the upstream secret, I see no change to the downstream secret. This extends to both editing the secret to change a label and deleting the upstream secret entirely. Does it have to do with excluding the namespace that the clone secret is derived from?

Steps to reproduce

Apply the below kyverno sync-secrets policy. Edit or delete the parent secret and observe that the child secrets are unaffected.

The policy:

# this is a Generate rule which creates new Kubernetes resources based on a
# policy and optionally keep them in sync. See more:
# https://kyverno.io/docs/writing-policies/generate/
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: sync-secrets
  annotations:
    policies.kyverno.io/title: Sync Secrets
    policies.kyverno.io/category: my-team
    policies.kyverno.io/subject: Secret
    policies.kyverno.io/minversion: 1.6.0
    policies.kyverno.io/description: >-
      Copy the wildcard secret into every namespace and keep it in sync.
spec:
  generateExisting: true
  background: false
  rules:
  - name: sync-wildcard-secret
    match:
      any:
      - resources:
          kinds:
            - Namespace
    exclude:
      any:
      - resources:
          namespaces:
          - kube-system
          - default
          - kube-public
          - kube-node-lease
          - kyverno
    generate:
      kind: Secret
      apiVersion: v1
      name: wildcard
      namespace: "{{ request.object.metadata.name }}"
      synchronize: true
      clone:
        namespace: kube-system
        name: wildcard

then either kubectl edit or kubectl delete the original secret and observe no change in the data inside the child secrets.

Expected behavior

Per https://kyverno.io/docs/writing-policies/generate/#clone-source I expected to see the downstream copies of the secret deleted.

Is it because the clone source of the secret is in a namespace that is being excluded?

The below logs are when deleting the child secrets where evidence is shown of the sync being respected. No logs are produced when the clone source is deleted.

Screenshots

No response

Kyverno logs

kyverno-background-controller-67d4c74d47-g8t49 controller {"level":"info","ts":"2024-06-27T16:06:06Z","logger":"background","caller":"generate/generate.go:101","msg":"start processing UR","name":"ur-c2wn7","policy":"sync-secrets","resource":"v1/Namespace//","ur":"ur-c2wn7","resourceVersion":"146331005"}
kyverno-background-controller-67d4c74d47-g8t49 controller {"level":"info","ts":"2024-06-27T16:06:06Z","logger":"background","caller":"generate/generate.go:101","msg":"start processing UR","name":"ur-s4fsh","policy":"sync-secrets","resource":"v1/Namespace//","ur":"ur-s4fsh","resourceVersion":"146331006"}
kyverno-background-controller-67d4c74d47-g8t49 controller {"level":"Level(-2)","ts":"2024-06-27T16:06:06Z","logger":"background","caller":"generate/generate.go:458","msg":"created generate target resource","name":"ur-s4fsh","policy":"sync-secrets","resource":"v1/Namespace//","target":"v1/Secret/example-system-global/wildcard"}
kyverno-background-controller-67d4c74d47-g8t49 controller {"level":"Level(-2)","ts":"2024-06-27T16:06:06Z","logger":"background","caller":"generate/generate.go:458","msg":"created generate target resource","name":"ur-c2wn7","policy":"sync-secrets","resource":"v1/Namespace//","target":"v1/Secret/example-system-global/wildcard"}

Slack discussion

No response

Troubleshooting

• I have read and followed the documentation AND the troubleshooting guide.
• I have searched other issues in this repository and mine is not recorded.

[chipzoller]

Does it have to do with excluding the namespace that the clone secret is derived from?

Yes, almost certainly.

[eitah]

the problem I'm trying to solve by excluding the namespace where the clone secret is is that Kyverno does not own the clone secret and it should not try to write to that namespace.

Is there another way besides rules.exclude to tell the below policy to ignore updates to the cloned secret's namespace?

      namespace: "{{ request.object.metadata.name }}"

An obvious workaround is to name the copy of the secret something else (i.e. wildcard-copy) but that's logistically complicated.

[chipzoller]

kube-system is excluded in webhooks by default, therefore whether it's listed in the exclude block in this policy is irrelevant. Kyverno must be able to "see" admission events on clone sources, and with it being excluded in the webhook this isn't possible. You can change the webhook exclusions either during install or on day2 by modifying Kyverno's ConfigMap.

[eitah]

Thanks! Please close, I've redirected this as a documentation fix. Appreciate the help!