Validation on managed resources

[rdavyd]

Hi,

I've created the following simple policy:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: validate-require-emptydir-size
  annotations:
    pod-policies.kyverno.io/autogen-controllers: "none"
spec:
  validationFailureAction: enforce
  background: true
  rules:
  - name: validate-require-emptydir-size
    match:
      resources:
        namespaces:
        - default
        kinds:
        - Pod
    validate:
      message: "You must set `sizeLimit` for volume type `emptyDir`."
      pattern:
        spec:
          =(volumes):
            - =(emptyDir):
                sizeLimit: "?*"

It does not work for Pods that are created by some controller, for example, deployment.
In logs I see the following:

I0406 07:44:24.883509       1 validation.go:89] EngineValidate "msg"="skip policy as direct changes to pods managed by workload controllers are not allowed" "kind"="Pod" "name"="vol-test-56ff5bdfbb-m5qtw" "namespace"="default" "policy"="validate-require-emptydir-size"

The question is if I can adjust this kyverno behavior to allow/disallow validation on managed resources?
Question is related to some mutating webhooks (non-kyverno) that mutate Pod manifests that are created by Deployment, daemonset etc. controllers. And I would like to validate that mutation.

[realshuting]

Hi @rdavyd - Kyverno already manages that for you, it generates rules for pod controllers automatically, see doc.

You can check these rules by kubectl get cpol validate-require-emptydir-size -o yaml. Since the policy is set to enforce, the resource (that violates the policy) creation should be blocked.

[rdavyd]

Actually I was migrating from gatekeeper and for me it was a notable difference - kyverno does not validate/mutate PODs that are managed by deployments, statefulsets etc.
You have to validate/mutate deployment, statefulset instead.
If I get it right this might be a useful note in the documentation.

[realshuting]

kyverno does not validate/mutate PODs that are managed by deployments, statefulsets etc.

Yes that's the case. For mutate policy, if you patch pods that are managed by pod controllers, the change will be reverted by these controllers.

This is what we have currently:

But creation of Pods directly is almost never done as it is considered an anti-pattern. Instead, Kubernetes has many higher-level controllers that directly or indirectly manage Pods, namely the Deployment, DaemonSet, StatefulSet, Job, and CronJob resources.

But I agree, this can be improved. Would you mind contributing a PR to our documentation?

[chipzoller]

I would be interested to hear your use case as to why you'd specifically want to mutate a Pod which was controlled by a higher-level abstraction.

[rdavyd]

I would be interested to hear your use case as to why you'd specifically want to mutate a Pod which was controlled by a higher-level abstraction.

It is a misguide on my part. In fact I do not really want to validate/mutate managed Pods.
I have a hashicorp vault-injector in the cluster which mutates Pods (not a Deployment itself) managed by Deployment by adding emptyDir volumes. And ran into situation when gatekeeper was denying validation because vault-injector does not specify sizeLimit for emptyDir, which I require to specify for applications.
When I started to evaluate kyverno this situation did not happen. And I ended up raising loglevel and reading code to be completely sure that it won't happen.
Now when I'm rethinking this, I find the approach to work only with higher-level abstractions completely valid.

[realshuting]

FYI, here's a sample policy to inject the sidecar container https://kyverno.io/policies/other/inject_sidecar_deployment/.