[Bug] verifyImages with autogen rules doesn't work for pods with 2+ containers

[seankhliao]

Kyverno Version

1.10.1

Kubernetes Version

1.26.x

Kubernetes Platform

KinD

Kyverno Rule Type

verifyImages

Description

When using verifyImages, with pod autogen rules, updates to Deployments (and other controllers) with 2+ containers fail, even though each image is independently verifiable.

Disabling autogen rules with allows the apply to succeed and updated pods to be created.

metadata:
  annotations:
    pod-policies.kyverno.io/autogen-controllers: none

In our production environment (EKS + Istio + ArgoCD), we get a slightly different error:

{"level":"info","ts":1687266173.007598,"logger":"klog","caller":"record/event.go:307","msg":"Event occurred","object":{"name":"verify-image-signature"},"fieldPath":"","kind":"ClusterPolicy","apiVersion":"kyverno.io/v1","type":"Warning","reason":"PolicyViolation","message":"Deployment default/xxx-agent: [autogen-verify-image] fail (blocked); kyverno.io/verify-images annotation cannot be changed"}

Though I've been unable to reproduce this specific error locally.

Steps to reproduce

1. kind create cluster --name kyverno --config kind.yaml
2. helm install kyverno kyverno/kyverno -n kyverno --create-namespace
3. kubectl apply -f deployment.k8s.yaml
   • this creates 3 single- deployments, proving each image is can be verified
   • also a double deployment with 2 containers which will break later
4. kubectl apply -f deployment-updated.k8s.yaml
   • updates one of the containers in double to a different, previously verified image

kind.yaml:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - role: control-plane
    image: kindest/node:v1.26.6@sha256:6e2d8b28a5b601defe327b98bd1c2d1930b49e5d8c512e1895099e4504007adb

clusterpolicy.k8s.yaml

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-image-signature
spec:
  background: false
  failurePolicy: Fail
  rules:
  - match:
      any:
      - resources:
          kinds:
          - Pod
    name: verify-image
    verifyImages:
    - attestors:
      - count: 1
        entries:
        - keys:
            publicKeys: |
              -----BEGIN PUBLIC KEY-----
              MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEMKLYTatU9CUsrA5Td6jXiZTolwsx
              HZKwYP5XkHhU436FGDD5Zi2nVFem6AbzXWHssIQRkAI3yJgKkB4J6Qe4OQ==
              -----END PUBLIC KEY-----
      imageReferences:
        - ghcr.io/seankhliao/*
      mutateDigest: false
      required: true
      verifyDigest: false
  validationFailureAction: Enforce
  webhookTimeoutSeconds: 30

deployment.k8s.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: single-6-3-3
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: single-6-3-3
  template:
    metadata:
      labels:
        app.kubernetes.io/name: single-6-3-3
    spec:
      containers:
        - name: podinfo
          image: ghcr.io/seankhliao/podinfo:6.3.3
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: single-6-3-4
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: single-6-3-4
  template:
    metadata:
      labels:
        app.kubernetes.io/name: single-6-3-4
    spec:
      containers:
        - name: podinfo
          image: ghcr.io/seankhliao/podinfo:6.3.4
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: single-6-3-5
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: single-6-3-5
  template:
    metadata:
      labels:
        app.kubernetes.io/name: single-6-3-5
    spec:
      containers:
        - name: podinfo
          image: ghcr.io/seankhliao/podinfo:6.3.5
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: double
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: double
  template:
    metadata:
      labels:
        app.kubernetes.io/name: double
    spec:
      containers:
        - name: podinfo-a
          image: ghcr.io/seankhliao/podinfo:6.3.3
        - name: podinfo-b
          image: ghcr.io/seankhliao/podinfo:6.3.4

deployment-updated.k8s.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: double
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: double
  template:
    metadata:
      labels:
        app.kubernetes.io/name: double
    spec:
      containers:
        - name: podinfo-a
          image: ghcr.io/seankhliao/podinfo:6.3.3
        - name: podinfo-b
          image: ghcr.io/seankhliao/podinfo:6.3.5

Expected behavior

Updated manifest to apply successfully

Screenshots

No response

Kyverno logs

main » k apply -f deployment-updated.k8s.yaml 
Error from server: error when applying patch:
{"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"apps/v1\",\"kind\":\"Deployment\",\"metadata\":{\"annotations\":{},\"name\":\"double\",\"namespace\":\"default\"},\"spec\":{\"selector\":{\"matchLabels\":{\"app.kubernetes.io/name\":\"double\"}},\"template\":{\"metadata\":{\"labels\":{\"app.kubernetes.io/name\":\"double\"}},\"spec\":{\"containers\":[{\"image\":\"ghcr.io/seankhliao/podinfo:6.3.3\",\"name\":\"podinfo-a\"},{\"image\":\"ghcr.io/seankhliao/podinfo:6.3.5\",\"name\":\"podinfo-b\"}]}}}}\n"}},"spec":{"template":{"spec":{"$setElementOrder/containers":[{"name":"podinfo-a"},{"name":"podinfo-b"}],"containers":[{"image":"ghcr.io/seankhliao/podinfo:6.3.5","name":"podinfo-b"}]}}}}
to:
Resource: "apps/v1, Resource=deployments", GroupVersionKind: "apps/v1, Kind=Deployment"
Name: "double", Namespace: "default"
for: "deployment-updated.k8s.yaml": error when patching "deployment-updated.k8s.yaml": admission webhook "validate.kyverno.svc-fail" denied the request: 

resource Deployment/default/double was blocked due to the following policies 

verify-image-signature:
  autogen-verify-image: unverified image ghcr.io/seankhliao/podinfo:6.3.3

---

I0623 08:49:27.824210       1 imageverifier.go:415] engine.verify "msg"="image attestors verification succeeded" "new.kind"="Deployment" "new.name"="double" "new.namespace"="default" "old.kind"="Deployment" "old.name"="double" "old.namespace"="default" "policy.apply"="All" "policy.name"="verify-image-signature" "policy.namespace"="" "requiredCount"=1 "rule.name"="autogen-verify-image" "verifiedCount"=1
I0623 08:49:27.824533       1 event.go:307] "Event occurred" object="verify-image-signature" fieldPath="" kind="ClusterPolicy" apiVersion="kyverno.io/v1" type="Normal" reason="PolicyApplied" message="Deployment default/double: pass"
I0623 08:49:27.832407       1 validation.go:103] webhooks/resource/validate "msg"="validation failed" "action"="Enforce" "clusterroles"=["cluster-admin","system:basic-user","system:discovery","system:public-info-viewer"] "failed rules"=["autogen-verify-image"] "gvk"={"group":"apps","version":"v1","kind":"Deployment"} "gvr"={"group":"apps","version":"v1","resource":"deployments"} "kind"="Deployment" "name"="double" "namespace"="default" "operation"="UPDATE" "policy"="verify-image-signature" "resource"="default/Deployment/double" "resource.gvk"={"Group":"apps","Version":"v1","Kind":"Deployment"} "roles"=null "uid"="e47a1fff-69ee-450f-80f7-2db4e2f4800a" "user"={"username":"kubernetes-admin","groups":["system:masters","system:authenticated"]}
I0623 08:49:27.832435       1 block.go:29] webhooks/resource/validate "msg"="blocking admission request" "action"="validate" "clusterroles"=["cluster-admin","system:basic-user","system:discovery","system:public-info-viewer"] "gvk"={"group":"apps","version":"v1","kind":"Deployment"} "gvr"={"group":"apps","version":"v1","resource":"deployments"} "kind"="Deployment" "name"="double" "namespace"="default" "operation"="UPDATE" "policy"="verify-image-signature" "resource"="default/Deployment/double" "resource.gvk"={"Group":"apps","Version":"v1","Kind":"Deployment"} "roles"=null "uid"="e47a1fff-69ee-450f-80f7-2db4e2f4800a" "user"={"username":"kubernetes-admin","groups":["system:masters","system:authenticated"]}
I0623 08:49:27.832501       1 handlers.go:139] webhooks/resource/validate "msg"="admission request denied" "clusterroles"=["cluster-admin","system:basic-user","system:discovery","system:public-info-viewer"] "gvk"={"group":"apps","version":"v1","kind":"Deployment"} "gvr"={"group":"apps","version":"v1","resource":"deployments"} "kind"="Deployment" "name"="double" "namespace"="default" "operation"="UPDATE" "resource.gvk"={"Group":"apps","Version":"v1","Kind":"Deployment"} "roles"=null "uid"="e47a1fff-69ee-450f-80f7-2db4e2f4800a" "user"={"username":"kubernetes-admin","groups":["system:masters","system:authenticated"]}
I0623 08:49:27.832643       1 event.go:307] "Event occurred" object="verify-image-signature" fieldPath="" kind="ClusterPolicy" apiVersion="kyverno.io/v1" type="Warning" reason="PolicyViolation" message="Deployment default/double: [autogen-verify-image] fail (blocked); unverified image ghcr.io/seankhliao/podinfo:6.3.3"

Slack discussion

No response

Troubleshooting

• I have read and followed the documentation AND the troubleshooting guide.
• I have searched other issues in this repository and mine is not recorded.

[welcome]

Thanks for opening your first issue here! Be sure to follow the issue template!

[eddycharly]

Thanks for reporting !

[eddycharly]

Only updates are affected by this bug, that's why disabling autogen workarounds the issue (pods are not updated, they are replaced).

[eddycharly]

Then annotation cannot be changed error message happens when something changes the annotation externally (only kyverno is allowed to do this).