The results of some of the rules in a policy are missing in their corresponding test.yaml.

[viveksahu26]

Description of the problem
The results of some of the rules in a policy are missing in their corresponding test.yaml. For example, In this policy, 3 rules are present in disallow_cri_sock_mount.yaml but if you look at test.yaml file, contains the results for 2 rules only.

To reproduce:-
Get the policy, resource and test.yaml file from here(https://github.com/kyverno/policies/tree/main/best-practices/disallow_cri_sock_mount).
Run the command kubectl kyverno apply disallow_cri_sock_mount.yaml -r resource.yaml --policy-report.

Policy-report summary:
error: 0
fail: 1
pass: 2
skip: 0
warn: 0

As we can see that, out of 3 rules, 2 rules are pass and one rule fails. But, in test.yaml, results for 2 rules are mention but results for one rule is missing.

Solution:-
So, finally, the conclusion is results of each rule mention in the policy should be present in the test.yaml file.

Originally posted by @viveksahu26 in #2039

[chipzoller]

Should you be assigned this instead, @viveksahu26 ?

[viveksahu26]

Let it be with @NoSkillGirl .

[NoSkillGirl]

@viveksahu26 - The test case is working as expected. I think you are little confused between the use case of apply and test command.

Considering the above case:
In apply we will get in total 3 result (pass, fail, skip, etc ...) as apply shows the final result of complete policy applied on a resource.
In test we will get only 2 results as test results shows the pass or fail for the cases that are mentioned in the test.yaml. For getting 3 results user should mention 3 test cases in test.yaml.

For more information check the documentation - https://kyverno.io/docs/kyverno-cli/

@chipzoller - please check it once, how clear is the above case is to the user while you are reviewing the documentation.

[chipzoller]

Honestly, the CLI documentation needs a major rework as there isn't enough information in either of these commands to give a complete enough explanation on what they do and how to use them.

[NoSkillGirl]

@chipzoller - Please try to mention the points as much you can. If you need any help about the functionality description, do let me know.

[NoSkillGirl]

update documentation for this issue - kyverno/website#212