installing kyverno as helm dependency fails when trying to use CustomResourceDefinition

[domruf]

Software version numbers

• Kubernetes version: 1.20.8
• Kubernetes platform (if applicable; ex., EKS, GKE, OpenShift): rancher
• Kyverno version: v2.1.3

Describe the bug
When tyring to install kyverno as a dependency in my own helm chart I get this error
Error: unable to build kubernetes objects from release manifest: unable to recognize "": no matches for kind "ClusterPolicy" in version "kyverno.io/v1"

To Reproduce
add

dependencies:
- name: kyverno
  version: v2.1.3
  repository: "https://kyverno.github.io/kyverno/"
- name: kyverno-policies
  version: v2.1.3
  repository: "https://kyverno.github.io/kyverno/"

to your Chart.yaml and call
helm upgrade --install --atomic myrelease .

Expected behavior
kyverno gets installed.
Instead I get the error mentioned above.

[welcome]

Thanks for opening your first issue here! Be sure to follow the issue template!

[kumar-mallikarjuna]

Hey @domruf - Seems like this is a more general problem with Helm 3. See:

• https://banzaicloud.com/blog/helm3-the-good-the-bad-and-the-ugly/#crd-as-a-first-class-citizen
• https://stackoverflow.com/a/60295868

Please correct me if I'm wrong.

[domruf]

@ojhaarjun1 thanks for your comment. I'm quite new to helm, so I'm not sure either. My understanding so far is, that CRDs should be put into the ./crds instead of the ./templates. That way they are installed before the 'normal' resources.
https://helm.sh/docs/chart_best_practices/custom_resource_definitions/#method-1-let-helm-do-it-for-you

So, I think ./templates/crds.yaml should be moved to ./crds/crds.yaml.
I'll try to experiment with this a bit and let you know what I find.

[JimBugwadia]

@domruf - the problem encountered with putting CRDs in the crds folder is that Helm does not upgrade them across version changes. Hence, they were moved to the templates.

There have been multiple attempts to achieve the right Helm UX for CRD handling, see: #2565, #2355, #2671. We are open to suggestions or guidance on what works best.

cc: @treydock

[treydock]

@domruf You can not have the helm install command do both the CRD as well as the things using the CRD when the CRD is a template. This why the kyverno-policies chart is separate, so it gets installed after the main kyverno chart. I would recommend splitting your dependencies so that you don't have both kyverno and kyverno-policies. You could install Kyverno with Helm then install a custom Helm chart that depends on kyverno-policies. I do something similar with this: https://github.com/OSC/osc-helm-charts/blob/main/charts/kyverno-policies/Chart.yaml

The other big project I know of that puts CRDs into the template directory is cert-manager, and for that reason you can't deploy cert-manager issuers in same Helm chart since they use the CRDs. In the case of cert-manager their CRDs are opt-in for Helm, but Kyverno does opt-out CRDs as a template to improve user experience for Helm users.

If it would be beneficial, one option is setting installCRDs Helm value to false for Kyverno chart and then can install CRDs separate. That is similar approach taken by cert-manager (Option 1): https://cert-manager.io/docs/installation/helm/#steps. If that seems like a useful approach to you, I could work on a pull request to make the CRD YAML more easily accessible to be installed via URL with kubectl or some other tool.

[kumar-mallikarjuna]

@domruf - Can we close this issue?

[domruf]

@ojhaarjun1 yes thanks