[Bug] Kyverno CLI test examples fail unless background: true is added or clusterRoles is removed from disallow-latest-tag policy

[menzenski]

Page link

https://kyverno.io/docs/kyverno-cli/usage/test/#examples

Description

I am new to Kyverno (not new to the Kubernetes ecosystem, though, if that is relevant) and am trying to set up some policies and test cases so that I can validate Kubernetes manifests outside of a cluster (the goal is to implement GitHub Actions pull request checks in a repository managed by ArgoCD).

I am working through the Examples section of the documentation on the kyverno test CLI command (https://kyverno.io/docs/kyverno-cli/usage/test/#examples) and I wasn't seeing the same output that the documentation showed.

I copied the YAML files provided by that documentation page (have them in a MWE git repo here too):

Contents of disallow_latest_tag.yaml:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-latest-tag
spec:
  validationFailureAction: Audit
  rules:
  - name: require-image-tag
    match:
      any:
      - resources:
          kinds:
          - Pod
        clusterRoles:
        - cluster-admin
    validate:
      message: "An image tag is required."  
      pattern:
        spec:
          containers:
          - image: "*:*"
  - name: validate-image-tag
    match:
      any:
      - resources:
          kinds:
          - Pod
    validate:
      message: "Using a mutable image tag e.g. 'latest' is not allowed."
      pattern:
        spec:
          containers:
          - image: "!*:latest"

Contents of resource.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: myapp-pod
  labels:
    app: myapp
spec: 
  containers:
  - name: nginx
    image: nginx:1.12

Contents of kyverno-test.yaml:

apiVersion: cli.kyverno.io/v1alpha1
kind: Test
metadata:
  name: disallow_latest_tag
policies:
  - disallow_latest_tag.yaml
resources:
  - resource.yaml
results:
  - policy: disallow-latest-tag
    rule: require-image-tag
    resource: myapp-pod
    kind: Pod
    result: pass
  - policy: disallow-latest-tag
    rule: validate-image-tag
    resource: myapp-pod
    kind: Pod
    result: pass

With these three files, the documentation shows kyverno test . returning this success response:

$ kyverno test .

Executing disallow_latest_tag...
applying 1 policy to 1 resource... 

│───│─────────────────────│────────────────────│───────────────────────│────────│
│ # │ POLICY              │ RULE               │ RESOURCE              │ RESULT │
│───│─────────────────────│────────────────────│───────────────────────│────────│
│ 1 │ disallow-latest-tag │ require-image-tag  │ default/Pod/myapp-pod │ Pass   │
│ 2 │ disallow-latest-tag │ validate-image-tag │ default/Pod/myapp-pod │ Pass   │
│───│─────────────────────│────────────────────│───────────────────────│────────│

Test Summary: 2 tests passed and 0 tests failed

However, when I run the same kyverno test . command, I get this failure response:

$ kyverno test .

WARNING: test file (kyverno-test.yaml) uses a deprecated schema that will be removed in 1.13
Loading test  ( kyverno-test.yaml ) ...
  Loading values/variables ...
  Loading policies ...
  Loading resources ...
  Loading exceptions ...
  Applying 1 policy to 1 resource ...
  Checking results ...

│────│─────────────────────│────────────────────│───────────────│────────│───────────│
│ ID │ POLICY              │ RULE               │ RESOURCE      │ RESULT │ REASON    │
│────│─────────────────────│────────────────────│───────────────│────────│───────────│
│ 1  │ disallow-latest-tag │ require-image-tag  │ Pod/myapp-pod │ Fail   │ Not found │
│ 2  │ disallow-latest-tag │ validate-image-tag │ Pod/myapp-pod │ Fail   │ Not found │
│────│─────────────────────│────────────────────│───────────────│────────│───────────│


Test Summary: 0 tests passed and 2 tests failed

Aggregated Failed Test Cases :
│────│─────────────────────│────────────────────│───────────────│────────│───────────│
│ ID │ POLICY              │ RULE               │ RESOURCE      │ RESULT │ REASON    │
│────│─────────────────────│────────────────────│───────────────│────────│───────────│
│ 1  │ disallow-latest-tag │ require-image-tag  │ Pod/myapp-pod │ Fail   │ Not found │
│ 2  │ disallow-latest-tag │ validate-image-tag │ Pod/myapp-pod │ Fail   │ Not found │
│────│─────────────────────│────────────────────│───────────────│────────│───────────│
Error: 2 tests failed

I searched the Kyverno discussions (link to search) and came across this comment, which suggested adding the -v=8 flag for more detailed output.

When I pass -v=8, I get an additional ERROR log message in the output:

2024-05-10T08:46:27-05:00	ERROR	kubectl-kyverno	test/test.go:150	skipping invalid policy	{"name": "disallow-latest-tag", "error": "only select variables are allowed in background mode. Set spec.background=false to disable background mode for this policy rule: invalid variable used at path: spec/rules[0]/match/any[0]/clusterRoles "}

Full output:

$ kyverno test . -v=8

WARNING: test file (kyverno-test.yaml) uses a deprecated schema that will be removed in 1.13
Loading test  ( kyverno-test.yaml ) ...
  Loading values/variables ...
  Loading policies ...
  Loading resources ...
  Loading exceptions ...
  Applying 1 policy to 1 resource ...
2024-05-10T08:46:27-05:00	ERROR	kubectl-kyverno	test/test.go:150	skipping invalid policy	{"name": "disallow-latest-tag", "error": "only select variables are allowed in background mode. Set spec.background=false to disable background mode for this policy rule: invalid variable used at path: spec/rules[0]/match/any[0]/clusterRoles "}
  Checking results ...

│────│─────────────────────│────────────────────│───────────────│────────│───────────│
│ ID │ POLICY              │ RULE               │ RESOURCE      │ RESULT │ REASON    │
│────│─────────────────────│────────────────────│───────────────│────────│───────────│
│ 1  │ disallow-latest-tag │ require-image-tag  │ Pod/myapp-pod │ Fail   │ Not found │
│ 2  │ disallow-latest-tag │ validate-image-tag │ Pod/myapp-pod │ Fail   │ Not found │
│────│─────────────────────│────────────────────│───────────────│────────│───────────│


Test Summary: 0 tests passed and 2 tests failed

Aggregated Failed Test Cases :
│────│─────────────────────│────────────────────│───────────────│────────│───────────│
│ ID │ POLICY              │ RULE               │ RESOURCE      │ RESULT │ REASON    │
│────│─────────────────────│────────────────────│───────────────│────────│───────────│
│ 1  │ disallow-latest-tag │ require-image-tag  │ Pod/myapp-pod │ Fail   │ Not found │
│ 2  │ disallow-latest-tag │ validate-image-tag │ Pod/myapp-pod │ Fail   │ Not found │
│────│─────────────────────│────────────────────│───────────────│────────│───────────│
Error: 2 tests failed

If I remove the clusterRoles property from the require-image-tag rule:

       - resources:
           kinds:
           - Pod
-        clusterRoles:
-        - cluster-admin
     validate:
       message: "An image tag is required."  
       pattern:

Then I get the same behavior (success) as the documentation example:

$ kyverno test . -v=8

WARNING: test file (kyverno-test.yaml) uses a deprecated schema that will be removed in 1.13
Loading test  ( kyverno-test.yaml ) ...
  Loading values/variables ...
  Loading policies ...
  Loading resources ...
  Loading exceptions ...
  Applying 1 policy to 1 resource ...
  Checking results ...

│────│─────────────────────│────────────────────│───────────────│────────│────────│
│ ID │ POLICY              │ RULE               │ RESOURCE      │ RESULT │ REASON │
│────│─────────────────────│────────────────────│───────────────│────────│────────│
│ 1  │ disallow-latest-tag │ require-image-tag  │ Pod/myapp-pod │ Pass   │ Ok     │
│ 2  │ disallow-latest-tag │ validate-image-tag │ Pod/myapp-pod │ Pass   │ Ok     │
│────│─────────────────────│────────────────────│───────────────│────────│────────│


Test Summary: 2 tests passed and 0 tests failed

There is still a difference between the output in the documentation and my local output, in that the documentation shows the default namespace in output (in the resource column), while my local output does not - that doesn't feel significant to me but wanted to mention it here regardless.

I also was curious about the spec.background=false bit in the error message, so searched for that and found the Background Scans documentation. Following that, I set background: false in the policy:

 metadata:
   name: disallow-latest-tag
 spec:
+  background: false
   validationFailureAction: Audit
   rules:
   - name: require-image-tag

And this also allows the kyverno test . command to pass (even if the clusterRoles property is present).

Expected behavior

I expect that when running the same commands on the same files provided in the documentation (using the same CLI version), that I get the same output.

Slack discussion

n/a

[welcome]

Thanks for opening your first issue here! Be sure to follow the issue template!