[BUG] CRDs are only loaded on startup and not refreshed

[MarcelMue]

Software version numbers
State the version numbers of applications involved in the bug.

• Kubernetes version: v1.21.1
• Kyverno version: 1.4.1

Describe the bug
Add CRD and mutation policy on this CRD while Kyverno is running -> policy always fails until Kyverno pod is restarted

To Reproduce
Steps to reproduce the behavior:

1. Start a kyverno pod in your cluster
2. Create a new CRD
3. Apply a policy which mutates based on the CRD

Expected behavior
The mutation policy should work and no errors should be emitted.

Additional context
The error log which happens:

I0702 08:35:40.615809       1 mutation.go:102] WebhookServer "msg"="validation error" "action"="mutate" "gvk"="infrastructure.cluster.x-k8s.io/v1alpha3, Kind=AWSClusterRoleIdentity" "operation"="CREATE" "resource"="AWSClusterRoleIdentity/test-cluster-4" "error"="pre-validation: couldn't find model , err: invalid kind" "policy"="aws-capi-policy"

After this log, the mutation rule is not triggered.

Restarting the kyverno pod leads to the expected behavior.

Resources for easy reproduction
CRD: https://raw.githubusercontent.com/kubernetes-sigs/cluster-api-provider-aws/main/config/crd/bases/infrastructure.cluster.x-k8s.io_awsclusterroleidentities.yaml

CR to apply:

apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
kind: AWSClusterRoleIdentity
metadata:
  labels:
    cluster.x-k8s.io/watch-filter: capi
    giantswarm.io/cluster: some-name
    cluster.x-k8s.io/cluster-name: some-name
  name: some-name
  namespace: default
spec:
  allowedNamespaces:
    list:
    - org-test
  roleARN: ""

Rule:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: aws-capi-policy
spec:
  background: false
  rules:
  - name: awsclusterroleidentity-default
    match:
      resources:
        kinds:
        - infrastructure.cluster.x-k8s.io/v1alpha3/AWSClusterRoleIdentity
        selector:
          matchLabels:
            cluster.x-k8s.io/watch-filter: capi
    mutate:
      patchesJson6902: |-
        - op: add
          path: "/spec/sourceIdentityRef/name"
          value: something
        - op: add
          path: "/spec/sourceIdentityRef/kind"
          value: AWSClusterControllerIdentity

[realshuting]

Thanks @MarcelMue, we'll investigate this issue and get back.

[MarcelMue]

@realshuting Has there been progress on this issue? We run into problems when we add new CRDs currently.

[realshuting]

No I wasn't able to work on this, will look into it tomorrow and get back to you.

[realshuting]

Hi @MarcelMue - looks like the issue was resolved in the latest main, you can verify with image tag v1.4.1-78-gacc9f665. Here's the result of my local test:

✗ k get awsclusterroleidentity some-name -o yaml | k neat
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha4
kind: AWSClusterRoleIdentity
metadata:
  annotations:
    policies.kyverno.io/patches: |
      awsclusterroleidentity-default.aws-capi-policy.kyverno.io: added /spec/sourceIdentityRef
  labels:
    cluster.x-k8s.io/cluster-name: some-name
    cluster.x-k8s.io/watch-filter: capi
    giantswarm.io/cluster: some-name
  name: some-name
spec:
  allowedNamespaces:
    list:
    - org-test
  roleARN: ""
  sourceIdentityRef:
    kind: AWSClusterControllerIdentity
    name: something

[realshuting]

BTW, if tag sourceIdentityRef does not exist in the resource, you can patch it with a single JSON patch:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: aws-capi-policy
spec:
  background: false
  rules:
  - name: awsclusterroleidentity-default
    match:
      resources:
        kinds:
        - infrastructure.cluster.x-k8s.io/v1alpha3/AWSClusterRoleIdentity
        selector:
          matchLabels:
            cluster.x-k8s.io/watch-filter: capi
    mutate:
      patchesJson6902: |-
        - op: add
          path: "/spec/sourceIdentityRef"
          value: {"kind": "AWSClusterControllerIdentity","name": "something"}

I'm closing this issue for now. Let me know if it does not work for you.

[MarcelMue]

Thanks for the heads up - will test this :)