Helm upgrade fails to install clustertriggerauthentications crd

[smarcu92]

Report

Hi,
Not sure if this is the correct place for this however, here we go...
Helm upgrade from 2.0.0 to 2.2.0 fails to install (clustertriggerauthentications.keda.sh crd)
We're running keda v2.0.0 installed via helm 3 on v1.19.6 (AKS)

• Installed using --set=installCRDs=true --set=podIdentity.activeDirectory.identity=<aad_identity>

Since the crd is missing pod fails to start.
(no matches for kind "ClusterTriggerAuthentication" in version "keda.sh/v1alpha1)

Expected Behavior

Missing components are installed and images are upgraded to the version specified.

Actual Behavior

Images get upgraded to the version specified.
Missing components are not installed, in this case clustertriggerauthentications.keda.sh is not installed.

If I manually install the missing crd, everything works as expected.

Steps to Reproduce the Problem

1. Ensure that keda crds are not already installed
2. helm install keda kedacore/keda -n keda --version=2.0.0 --set=installCRDs=true --set=podIdentity.activeDirectory.identity=<aad_identity> --debug
3. helm upgrade helm install keda kedacore/keda -n keda --version=2.2.0 --debug
4. check pod logs, check crd

Logs from KEDA operator

2021-03-26T14:47:51.962Z	INFO	controller-runtime.metrics	metrics server is starting to listen	{"addr": ":8080"}
2021-03-26T14:47:51.965Z	INFO	controllers.ScaledObject	Running on Kubernetes 1.19	{"version": "v1.19.6"}
2021-03-26T14:47:51.965Z	INFO	setup	Starting manager
2021-03-26T14:47:51.965Z	INFO	setup	KEDA Version: 2.2.0
2021-03-26T14:47:51.965Z	INFO	setup	Git Commit: 6dc4c3524ba6278276cd1338cc7e4e9a7dd88963
2021-03-26T14:47:51.965Z	INFO	setup	Go Version: go1.15.6
2021-03-26T14:47:51.965Z	INFO	setup	Go OS/Arch: linux/amd64
I0326 14:47:51.966071       1 leaderelection.go:243] attempting to acquire leader lease keda/operator.keda.sh...
2021-03-26T14:47:51.966Z	INFO	controller-runtime.manager	starting metrics server	{"path": "/metrics"}
I0326 14:48:09.381707       1 leaderelection.go:253] successfully acquired lease keda/operator.keda.sh
2021-03-26T14:48:09.381Z	INFO	controller	Starting EventSource	{"reconcilerGroup": "keda.sh", "reconcilerKind": "ScaledJob", "controller": "scaledjob", "source": "kind source: /, Kind="}
2021-03-26T14:48:09.381Z	INFO	controller	Starting EventSource	{"reconcilerGroup": "keda.sh", "reconcilerKind": "ClusterTriggerAuthentication", "controller": "clustertriggerauthentication", "source": "kind source: /, Kind="}
2021-03-26T14:48:09.381Z	INFO	controller	Starting EventSource	{"reconcilerGroup": "keda.sh", "reconcilerKind": "TriggerAuthentication", "controller": "triggerauthentication", "source": "kind source: /, Kind="}
2021-03-26T14:48:09.381Z	INFO	controller	Starting EventSource	{"reconcilerGroup": "keda.sh", "reconcilerKind": "ScaledObject", "controller": "scaledobject", "source": "kind source: /, Kind="}
2021-03-26T14:48:09.482Z	INFO	controller	Starting Controller	{"reconcilerGroup": "keda.sh", "reconcilerKind": "ScaledJob", "controller": "scaledjob"}
2021-03-26T14:48:10.285Z	ERROR	controller-runtime.source	if kind is a CRD, it should be installed before calling Start	{"kind": "ClusterTriggerAuthentication.keda.sh", "error": "no matches for kind \"ClusterTriggerAuthentication\" in version \"keda.sh/v1alpha1\""}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.5/pkg/internal/controller/controller.go:143
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.5/pkg/internal/controller/controller.go:184
sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable.func1
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.5/pkg/manager/internal.go:676
2021-03-26T14:48:10.285Z	INFO	controller	Starting workers	{"reconcilerGroup": "keda.sh", "reconcilerKind": "ScaledJob", "controller": "scaledjob", "worker count": 1}
2021-03-26T14:48:10.285Z	INFO	controller	Stopping workers	{"reconcilerGroup": "keda.sh", "reconcilerKind": "ScaledJob", "controller": "scaledjob"}

KEDA Version

2.2.0

Kubernetes Version

1.19

Platform

Microsoft Azure

Scaler Details

Azure Service Bus

[herrbpl]

Have same issue

[tomkerkhove]

Might be related to helm/helm#6581 & helm/helm#8668

[tomkerkhove]

As a mitigation, you can manually install the CRD (https://github.com/kedacore/charts/blob/master/keda/crds/clustertriggerauthentications.yaml).

[tomkerkhove]

As per: https://helm.sh/docs/chart_best_practices/custom_resource_definitions/#install-a-crd-declaration-before-using-the-resource

There is no support at this time for upgrading or deleting CRDs using Helm. This was an explicit decision after much community discussion due to the danger for unintentional data loss. Furthermore, there is currently no community consensus around how to handle CRDs and their lifecycle. As this evolves, Helm will add support for those use cases.

It looks like we'll need to find a way around this.

[MattJeanes]

This should be noted in upgrade notes as when installing through Helm this is technically a breaking change

Workaround: kubectl apply -f https://raw.githubusercontent.com/kedacore/charts/master/keda/crds/clustertriggerauthentications.yaml

[tomkerkhove]

I've opened #140 to fix this so that they are always added and I'll update the docs asap.

Sorry for the inconvenience. I wish Helm was handling this better.

[smarcu92]

Hi @tomkerkhove,

Hope you're well.

I can confirm the workaround works since I tested it before creating the ticket.

However, in this specific case we're not actually talking about a CRD upgrade using Helm since the actual CRD is missing to begin with. So my exception would be that if the CRD is not installed and keda has a hard dependency on it, exactly the situation we're facing now , it should be installed.

I would also argue that the user should have the possibility of forcing the upgrade of a CRD through Helm, using a specific switch (with full accountability of course on his end, knowing that the process is not supported and may potentially cause data lose).

Anyway that's just my point of view.

Let me know what you think.

Thanks for all your help and support.
Be safe!

[tomkerkhove]

Fully agree, but apparently Helm doesn't agree on that (see above).

I'm working on a fix for this in #141

[tomkerkhove]

This is fixed and available: https://github.com/kedacore/charts/releases/tag/v2.2.1

Sorry about this.

[micaelgallego]

Install doc still references to this issue. But I assume this issue is fixed and workaround is not longer needed, right?

[tomkerkhove]

Yes, docs only refer to v2.1 or below. v2.2 normally doesn't point here.

[maxisam]

Yes, docs only refer to v2.1 or below. v2.2 normally doesn't point here.

2.2 still points to here

[tomkerkhove]

Thanks for letting me know @maxisam!