Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Helm upgrade fails to install clustertriggerauthentications crd #139

Closed
smarcu92 opened this issue Mar 26, 2021 · 13 comments
Closed

Helm upgrade fails to install clustertriggerauthentications crd #139

smarcu92 opened this issue Mar 26, 2021 · 13 comments

Comments

@smarcu92
Copy link

Report

Hi,
Not sure if this is the correct place for this however, here we go...
Helm upgrade from 2.0.0 to 2.2.0 fails to install (clustertriggerauthentications.keda.sh crd)
We're running keda v2.0.0 installed via helm 3 on v1.19.6 (AKS)

  • Installed using --set=installCRDs=true --set=podIdentity.activeDirectory.identity=<aad_identity>

Since the crd is missing pod fails to start.
(no matches for kind "ClusterTriggerAuthentication" in version "keda.sh/v1alpha1)

Expected Behavior

Missing components are installed and images are upgraded to the version specified.

Actual Behavior

Images get upgraded to the version specified.
Missing components are not installed, in this case clustertriggerauthentications.keda.sh is not installed.

If I manually install the missing crd, everything works as expected.

Steps to Reproduce the Problem

  1. Ensure that keda crds are not already installed
  2. helm install keda kedacore/keda -n keda --version=2.0.0 --set=installCRDs=true --set=podIdentity.activeDirectory.identity=<aad_identity> --debug
  3. helm upgrade helm install keda kedacore/keda -n keda --version=2.2.0 --debug
  4. check pod logs, check crd

Logs from KEDA operator

2021-03-26T14:47:51.962Z	INFO	controller-runtime.metrics	metrics server is starting to listen	{"addr": ":8080"}
2021-03-26T14:47:51.965Z	INFO	controllers.ScaledObject	Running on Kubernetes 1.19	{"version": "v1.19.6"}
2021-03-26T14:47:51.965Z	INFO	setup	Starting manager
2021-03-26T14:47:51.965Z	INFO	setup	KEDA Version: 2.2.0
2021-03-26T14:47:51.965Z	INFO	setup	Git Commit: 6dc4c3524ba6278276cd1338cc7e4e9a7dd88963
2021-03-26T14:47:51.965Z	INFO	setup	Go Version: go1.15.6
2021-03-26T14:47:51.965Z	INFO	setup	Go OS/Arch: linux/amd64
I0326 14:47:51.966071       1 leaderelection.go:243] attempting to acquire leader lease keda/operator.keda.sh...
2021-03-26T14:47:51.966Z	INFO	controller-runtime.manager	starting metrics server	{"path": "/metrics"}
I0326 14:48:09.381707       1 leaderelection.go:253] successfully acquired lease keda/operator.keda.sh
2021-03-26T14:48:09.381Z	INFO	controller	Starting EventSource	{"reconcilerGroup": "keda.sh", "reconcilerKind": "ScaledJob", "controller": "scaledjob", "source": "kind source: /, Kind="}
2021-03-26T14:48:09.381Z	INFO	controller	Starting EventSource	{"reconcilerGroup": "keda.sh", "reconcilerKind": "ClusterTriggerAuthentication", "controller": "clustertriggerauthentication", "source": "kind source: /, Kind="}
2021-03-26T14:48:09.381Z	INFO	controller	Starting EventSource	{"reconcilerGroup": "keda.sh", "reconcilerKind": "TriggerAuthentication", "controller": "triggerauthentication", "source": "kind source: /, Kind="}
2021-03-26T14:48:09.381Z	INFO	controller	Starting EventSource	{"reconcilerGroup": "keda.sh", "reconcilerKind": "ScaledObject", "controller": "scaledobject", "source": "kind source: /, Kind="}
2021-03-26T14:48:09.482Z	INFO	controller	Starting Controller	{"reconcilerGroup": "keda.sh", "reconcilerKind": "ScaledJob", "controller": "scaledjob"}
2021-03-26T14:48:10.285Z	ERROR	controller-runtime.source	if kind is a CRD, it should be installed before calling Start	{"kind": "ClusterTriggerAuthentication.keda.sh", "error": "no matches for kind \"ClusterTriggerAuthentication\" in version \"keda.sh/v1alpha1\""}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.5/pkg/internal/controller/controller.go:143
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.5/pkg/internal/controller/controller.go:184
sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable.func1
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.5/pkg/manager/internal.go:676
2021-03-26T14:48:10.285Z	INFO	controller	Starting workers	{"reconcilerGroup": "keda.sh", "reconcilerKind": "ScaledJob", "controller": "scaledjob", "worker count": 1}
2021-03-26T14:48:10.285Z	INFO	controller	Stopping workers	{"reconcilerGroup": "keda.sh", "reconcilerKind": "ScaledJob", "controller": "scaledjob"}

KEDA Version

2.2.0

Kubernetes Version

1.19

Platform

Microsoft Azure

Scaler Details

Azure Service Bus

@herrbpl
Copy link

herrbpl commented Apr 7, 2021

Have same issue

@tomkerkhove tomkerkhove transferred this issue from kedacore/keda Apr 8, 2021
@tomkerkhove
Copy link
Member

tomkerkhove commented Apr 8, 2021

Might be related to helm/helm#6581 & helm/helm#8668

@tomkerkhove
Copy link
Member

As a mitigation, you can manually install the CRD (https://github.com/kedacore/charts/blob/master/keda/crds/clustertriggerauthentications.yaml).

@tomkerkhove
Copy link
Member

As per: https://helm.sh/docs/chart_best_practices/custom_resource_definitions/#install-a-crd-declaration-before-using-the-resource

There is no support at this time for upgrading or deleting CRDs using Helm. This was an explicit decision after much community discussion due to the danger for unintentional data loss. Furthermore, there is currently no community consensus around how to handle CRDs and their lifecycle. As this evolves, Helm will add support for those use cases.

It looks like we'll need to find a way around this.

@MattJeanes
Copy link

This should be noted in upgrade notes as when installing through Helm this is technically a breaking change

Workaround: kubectl apply -f https://raw.githubusercontent.com/kedacore/charts/master/keda/crds/clustertriggerauthentications.yaml

@tomkerkhove
Copy link
Member

I've opened #140 to fix this so that they are always added and I'll update the docs asap.

Sorry for the inconvenience. I wish Helm was handling this better.

@smarcu92
Copy link
Author

Hi @tomkerkhove,

Hope you're well.

I can confirm the workaround works since I tested it before creating the ticket.

However, in this specific case we're not actually talking about a CRD upgrade using Helm since the actual CRD is missing to begin with. So my exception would be that if the CRD is not installed and keda has a hard dependency on it, exactly the situation we're facing now , it should be installed.

I would also argue that the user should have the possibility of forcing the upgrade of a CRD through Helm, using a specific switch (with full accountability of course on his end, knowing that the process is not supported and may potentially cause data lose).

Anyway that's just my point of view.

Let me know what you think.

Thanks for all your help and support.
Be safe!

@tomkerkhove
Copy link
Member

Fully agree, but apparently Helm doesn't agree on that (see above).

I'm working on a fix for this in #141

@tomkerkhove
Copy link
Member

This is fixed and available: https://github.com/kedacore/charts/releases/tag/v2.2.1

Sorry about this.

@micaelgallego
Copy link

Install doc still references to this issue. But I assume this issue is fixed and workaround is not longer needed, right?

@tomkerkhove
Copy link
Member

Yes, docs only refer to v2.1 or below. v2.2 normally doesn't point here.

@maxisam
Copy link

maxisam commented May 3, 2021

Yes, docs only refer to v2.1 or below. v2.2 normally doesn't point here.

2.2 still points to here

@tomkerkhove
Copy link
Member

Thanks for letting me know @maxisam!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants