CRDs not being applied correctly

[lachie83]

I'm seeing the following when installing the chart in this PR helm/charts#2369

$ helm install --name istio . --namespace istio-system
Error: unable to decode "": no kind "CustomResourceDefinition" is registered for version "apiextensions.k8s.io/v1beta1"

If I render the files with CRDs manually using helm template and install using kubectl create -f they work just fine.

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.6", GitCommit:"4bc5e7f9a6c25dc4c03d4d656f2cefd21540e28c", GitTreeState:"clean", BuildDate:"2017-09-15T08:51:09Z", GoVersion:"go1.9", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.5", GitCommit:"17d7182a7ccbb167074be7a87f0a68bd00d58d97", GitTreeState:"clean", BuildDate:"2017-08-31T08:56:23Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
$ helm version
Client: &version.Version{SemVer:"v2.6.1", GitCommit:"bbc1f71dc03afc5f00c6ac84b9308f8ecb4f39ac", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.6.1", GitCommit:"bbc1f71dc03afc5f00c6ac84b9308f8ecb4f39ac", GitTreeState:"clean"} 

[redbaron]

Look at prometheus-operator, helm is not waiting for third party resource to become fully registered, therefore one off install hook is necessary

[lachie83]

In this chart the services aren't creating the CRDs. The chart itself creates the CRDs. I've used hooks to order the creation but Helm is having trouble applying them to Kubernetes

[bacongobbler]

Is this a cascading failure from #2995? Let's try the proposed fix in that issue first and see if that fixes things.

[lachie83]

This didn't help. I believe there's some funniness when using CRDs/TPRs with hooks. #2680

[thomastaylor312]

Related: #2925

[thomastaylor312]

As a note, we are going to pull the ordering part of this into 2.7

[thomastaylor312]

Pulling this out of 2.7 in favor of #2925

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[thomastaylor312]

/lifecycle frozen

[benjigoldberg]

Whats the status of this? Is ordered support for CRDs in the roadmap for a future release or being worked on?

[bacongobbler]

This hasn't been slated into a release and nobody's working on this AFAIK. The root issue here is that we need some way to figure out how to install CRDs before beginning to validate the rest of the chart. It's a substantial change to the existing workflow, but it is possible to accomplish this in Helm 2 if someone wants to tackle the issue.

[benjigoldberg]

@bacongobbler thanks for the reply! I'm not sure I'm necessarily the right person to spearhead the effort, but I love helm and would love to help if possible. I think given the direction of the community right now and all the talk at Kubecon, this seems like something that could be a huge win for the tool. Do you guys have community/SIG meetings on zoom? I can try and drop by the next one and learn more and see how/where I can be of help.

[whereisaaron]

TPR/CRDs are a particular issue that often necessitates some ordering control. But there are other ordering issues for umbrella charts with many sub-charts. It would be nice to have a simple and general ordering mechanism, rather than a special behavior for TPR/CRD. As identified here, it is critical to consider when validation occurs.

One possible approach would to allow weights to be applied to subcharts in requirements.yaml, with the resources for charts of the same weight being sorted together with the current algorithm, validated together, and applied together. All weights, including the parent chart would be zero by default. Negative weights would install before the parent chart, positive weights would install after.

Deletions would follow the same grouping in reverse, deleting resources for equally weighted groups of parent and/or sub-charts, from highest weight to lowest weight.

With this optional feature the processing of all existing charts (with no weights) would be unaffected - for both deployment order, validation behavior, and deletion. Only people who really need ordering would specify any weights for requirements.yaml.

This mechanism would enable TPR/CRDs to be applied first, simply by including them in a lower weight chart (parent or sub-chart) than the charts (parent and/or sub-charts) that rely on those TPR/CRDs.

Considerations:

1. Would tiller need to wait for the previous deployed group to be successfully registered and/or running? If that is needed, chart authors could use hooks to e.g. delay until CRDs have fully registered. And possibly the current '--wait' mechanism could apply to each group of resources? Could '--wait' also include checking that TPR/CRDs have registered in the API?
2. What will the partial success/clean-up behaviour be? Since each group of charts of a weight (parent and/or sub-charts) are validated and applied before the next group can be validated, the subsequent groups might not validate or fail to apply.
3. This proposal is basically the current sort, validate and apply process, except applied in multiple rounds. You could simulate the same effect with multiple separate Helm chart packages, shared values files, and a script to apply them in the desired order. But that means house dependencies like TPR/CRD in separate chart packages.
4. The expectation is 90% of charts will continue to have no weights at all. Some charts with TPR/CRD would have two weights in the whole chart. And some complicated umbrella charts might need a handful of different weights.
5. Numeric weights are easy and flexible, but not super human friendly. Perhaps 'named' weights would be a better way to go. Charts could 'weighted' in requirements.yaml with just the labels "first" or "last" or not labelled that equates to weights of -1, 1 and 0 respectively (or -MaxInt, MaxInt, and 0 respectively). With only huge umbrella charts needing to resort to number labels.
6. A limitation of this approach is that to achieve ordering you need to have at least one (embedded) subchart. So a minimal chart with TPR/CRD would have an embedded subchart.
7. I know squat about the Helm code, does the above have complete show-stoppers that make it impossible or an unreasonably complicated way forward?

[luisyonaldo]

I think issue is just with helm validation. Instead of introduce weighs for requirements, is possible just to validate on installation level with communication with tiller instead of a pre installation step? At least just for CRD. With the introduction of operator in kubernetes, CRD are installed by the operator itself and not by the helm chart, which make still no possible to use the requirements weighs you suggest.

Is something like a registration step possible? That you can defined a job or something that validates the CRD installation using helm hooks.

In my opinion the approach should be as stated in the helm documentation

INSTALL A TPR DECLARATION BEFORE USING THE RESOURCE

Which can be done by installing the CRD as pre-install hook. With his the ordered is correct, and you can use hook weights for a more controlled ordering. Issue is still with helm validation, which will fail if it detects a custom resource that depends on a CRD that is being installed in the same install (even if the order is right, so CRD will be installed before the customer resource).

[kfox1111]

and certmanager

[paktek123]

I get same thing as @fiunchinho but for heptio-ark when trying to upgrade/install

[dabelenda]

I am using helm in an OpenShift setup, and trying to manage everything using helm hitting ordering issues very hard...

In OpenShift there is a CRD called Project, which is a complete replacement of Namespace (it then creates a "virtual" Namespace object Read-Only), thus when I try creating a Project with Quotas, and other official objects from k8s the ordering is messed up.

I think a good way to let users control ordering with CRDs, would be to, instead of having a static ordering like defined here https://github.com/helm/helm/blob/release-2.13/pkg/tiller/kind_sorter.go#L29 use an integer priority to each object and allow users to set a priority for Kinds in metadata of the chart or something.

[whereisaaron]

@dabelenda I've suggested elsewhere that subcharts in requirements.yaml could be weighted/ordered. Then you could split deployments into phases and control the order at that level?
#1780 (comment)

It would be nice to be able to e.g. apply weights to subcharts in requirements.yaml, with charts of the same weight being sorted together with the current algorithm. All weights, including parent chart would be zero by default. Negative weights would install before the parent chart, positive weights would install after.

With this optional feature the processing of all existing charts would be unaffected. And only people who really need ordering would specify any weights for requirements.yaml.

[dabelenda]

I am "only" creating the Project, Quotas, and RoleBindings here.
The need to create a dummy subchart to only create a Project and then apply Quotas and RoleBindings is weird.
Every single CRD that have dependencies will have to be in different subcharts. It would creates hard-to-understand charts if the dependency tree is non trivial.

I think of Prometheus-Operator Prometheus ServiceMonitoring etc

[Ciantic]

I'm using helm3 and experience the same problem sporadically, this time with istio resources:

Error: apiVersion "config.istio.io/v1alpha2" in istio/charts/mixer/templates/config.yaml is not available

It works occasionally, but most often it just throws me that error. It doesn't happen always.

[JulienBreux]

Error

Error: apiVersion "config.istio.io/v1alpha2" in istio/charts/mixer/templates/config.yaml is not available

Versions

• Helm: version.BuildInfo{Version:"v3.0.0-beta.3", GitCommit:"5cb923eecbe80d1ad76399aee234717c11931d9a", GitTreeState:"clean", GoVersion:"go1.12.9"}
• Istio files: https://github.com/istio/istio/tree/1.3.0/install/kubernetes/helm/istio-init

[thomastaylor312]

Hey @Ciantic and @JulienBreux so this is because your local types/discovery cache is stale after you install a CRD. We have handled this with the new crds/ directory installation in Helm 3. It will invalidate the cache for you. This is due to the way Kubernetes discovery caching works and not due to Helm itself per se. To get around it, you can delete ~/.kube/cache/discovery (or append the cluster name to the end of the path to just remove its specific cache) in between running the install of Istio init and installing Istio itself

[Ciantic]

@thomastaylor312 it doesn't work. I guess because the main istio chart also tries to install crds?

helm repo add istio.io https://storage.googleapis.com/istio-release/releases/1.3.0/charts/
helm repo update

kubectl create namespace istio-system

# Install crds
helm install istio-init istio.io/istio-init --namespace istio-system

# See https://github.com/helm/helm/issues/2994 for the deleting cache
rm -rf ~/.kube/cache/discovery/

sleep 3s # This really is installation step!

# Install istio
helm install istio istio.io/istio \
    --namespace istio-system \
    --set gateways.istio-ingressgateway.type=NodePort

Throws error:

Error: apiVersion "config.istio.io/v1alpha2" in istio/charts/mixer/templates/config.yaml is not available

Only thing that works 100% time is normal kubectl apply -f method.

I want to note that above script works sometimes but usually not. So something funny and non-deterministic there is.

[bacongobbler]

Probably because support for the crd-install hook was dropped in Helm 3 in favour of a crd directory.

[Ciantic]

You are right. Apparently istio can't be installed with helm3, and probably never will be, because they are working on a different installer. See istio/istio#17167

[bacongobbler]

closing as resolved with the crds directory. For further discussion on CRDs and Helm, please see #5871. Thanks.