helm-operator - failed to install release: rendered manifests contain a resource that already exists

[esara]

Bug Report

What did you do?
We have started building operators with v0.8.x, which in the case of the helm-operator was creating a helm2 release, later/automatically upgraded to a helm3 release with v0.13.x - v0.17.x based operators.
We started to have problems as we further upgraded the same operator/custom resource from v0.17.x to a v0.18.x based helm operator.

Specifically custom resources created with a helm2 operator, the helm2 release is named as (instead of just plain RELEASENAME)
$ kubectl get secrets RELEASENAME-84bxdj3wmek999m7mr3hnknlp.v1
NAME TYPE DATA AGE
RELEASENAME-84bxdj3wmek999m7mr3hnknlp.v1 Opaque 1 17s

What did you expect to see?
after upgrading the operator to a helm3 operator (1.17.1 operator sdk), it automatically upgrades to helm3
$ kubectl logs -f t8c-operator-75f4d6f79-tt927
{"level":"info","ts":1593364045.881045,"logger":"helm.controller","msg":"Updated release","namespace":"turbonomic","name":"xl-release","apiVersion":"charts.helm.k8s.io/v1alpha1","kind":"Xl","release":"xl-release-84bxdj3wmek999m7mr3hnknlp","force":false}
$ helm list -A
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
RELEASENAME-84bxdj3wmek999m7mr3hnknlp turbonomic 5 2020-06-28 17:07:24.960187627 +0000 UTC deployed xl-1.6.0 7.22
$ kubectl get secrets sh.helm.release.v1.RELEASENAME-84bxdj3wmek999m7mr3hnknlp.v5
NAME TYPE DATA AGE
sh.helm.release.v1.RELEASENAME-84bxdj3wmek999m7mr3hnknlp.v5 helm.sh/release.v1 1 88s
$ kubectl logs -f t8c-operator-75f4d6f79-tt927
{"level":"info","ts":1593364047.209766,"logger":"helm.controller","msg":"Reconciled release","namespace":"turbonomic","name":"xl-release","apiVersion":"charts.helm.k8s.io/v1alpha1","kind":"Xl","release":"xl-release-84bxdj3wmek999m7mr3hnknlp"}

What did you see instead? Under which circumstances?
but if I upgrade to the 1.18.x based helm operator, it fails to apply any changes, with
$ kubectl logs -f t8c-operator-6988fd7b6c-ls55s
{"level":"error","ts":1593364226.4781153,"logger":"helm.controller","msg":"Release failed","namespace":"turbonomic","name":"xl-release","apiVersion":"charts.helm.k8s.io/v1alpha1","kind":"Xl","release":"xl-release","error":"failed to install release: rendered manifests contain a resource that already exists. Unable to continue with install: PersistentVolumeClaim "rsyslog-auditlogdata" in namespace "turbonomic" exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: missing key "meta.helm.sh/release-name": must be set to "xl-release"; annotation validation error: missing key "meta.helm.sh/release-namespace": must be set to "turbonomic"","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\tpkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\ngithub.com/operator-framework/operator-sdk/pkg/helm/controller.HelmOperatorReconciler.Reconcile\n\tsrc/github.com/operator-framework/operator-sdk/pkg/helm/controller/reconcile.go:181\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:256\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\tpkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\tpkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\tpkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\tpkg/mod/k8s.io/apimachinery@v0.18.2/pkg/util/wait/wait.go:90"}

the actual helm chart used by the operator is the same…

Environment

• operator-sdk version:
  upgrading from v0.17.x to v0.18.x

• Kubernetes version information:
  $ kubectl version
  Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-16T11:56:40Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"darwin/amd64"}
  Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.10", GitCommit:"575467a0eaf3ca1f20eb86215b3bde40a5ae617a", GitTreeState:"clean", BuildDate:"2019-12-11T12:32:32Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}

and

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-16T11:56:40Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.3+a637491", GitCommit:"a637491", GitTreeState:"clean", BuildDate:"2020-06-05T15:48:59Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes cluster kind:
  OCP4 and/or upstream k8s (same results)

• Are you writing your operator in ansible, helm, or go?
  helm-operator published at https://github.com/turbonomic/t8c-install/tree/master/operator

Possible Solution
tried to do helm/helm#7649 but it deleted and recreated the resources and failed anyway

Additional context
also looked at helm/helm#8078 but this is a different problem, the helm chart is valid and installs fine (with both helm3 and helm2) and upgrades fine with v0.17.x (or earlier)

[camilamacedo86]

Hi @esara,

Following my test with your project and the SDK 0.18.2 version on minikube. Then, please check the migration guide for 0.18.x and that was removed the automatic migration of helm releases from v2 to v3 and beware that was removed the support for deprecated helm release naming scheme.

My tests with your project
Following the steps:

• Clone project https://github.com/turbonomic/t8c-install/tree/master/operator
• Update its Dockerfile to use 0.18.2 :

$ cat build/Dockerfile 
FROM quay.io/operator-framework/helm-operator:v0.18.2

• See that the release was made successfully and I did not check the error raised:

 $ kubectl get all
NAME                                READY   STATUS    RESTARTS   AGE
pod/t8c-operator-6797549c59-ffbxx   1/1     Running   0          7m21s

NAME                           TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
service/kubernetes             ClusterIP   10.96.0.1       <none>        443/TCP             26h
service/t8c-operator-metrics   ClusterIP   10.96.227.240   <none>        8383/TCP,8686/TCP   7m10s

NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/t8c-operator   1/1     1            1           7m21s

NAME                                      DESIRED   CURRENT   READY   AGE
replicaset.apps/t8c-operator-6797549c59   1         1         1       7m21s

 $ kubectl logs deployment.apps/t8c-operator 
...
{"level":"info","ts":1593691871.6366103,"logger":"helm.controller","msg":"Reconciled release","namespace":"default","name":"xl-release","apiVersion":"charts.helm.k8s.io/v1alpha1","kind":"Xl","release":"xl-release"}

$ kubectl describe XL xl-release
....
Status:
  Conditions:
    Last Transition Time:  2020-07-02T12:05:04Z
    Status:                True
    Type:                  Initialized
    Last Transition Time:  2020-07-02T12:05:05Z
    Reason:                InstallSuccessful
    Status:                True
    Type:                  Deployed
  Deployed Release:
    Manifest:  ---
...
kind: ConfigMap
metadata:
  name: global-properties-xl-release
  labels:
    app.kubernetes.io/name: xl
    app.kubernetes.io/instance: xl-release
    app.kubernetes.io/managed-by: Helm # See the label 
...

Regards:

tried to do helm/helm#7649 but it deleted and recreated the resources and failed anyway

Just to share, according to the helm/helm#7697 (comment) the error faced can be solved by adding the labels and using the Helm version >= v3.2.0 and I did not face the same by installing all from scratch as described above with 0.18.2. The SDK release 0.18.2 is using v3.2.0. The master branch is with v3.2.4 but both are no longer supporting the automigration of Helm v2 releases.

However, I'd recommend always check the behaviour with helm directly if possible too, since SDK uses Helm. Note that if the Helm version used by SDK has a bug then consequently it can impact the operator project since needs to be solved there. Could you please let us know if you are able to reproduce the same with Helm? If yes, could you please raise an issue there and ping us here with the link? When that get solved we will be able to use a new Helm release to get the fix for SDK as well.

[joelanford]

As @camilamacedo86 mentioned, I think the issue here is that in v0.18.x, the helm-operator base image no longer supports the legacy UID-based release name.

You should be able to avoid downtime by doing the following (I just double-checked this on a simple nginx-operator to confirm):

1. Stop the operator
2. Using the helm v3.2+ CLI, run helm install <cr-name> <chart> --values <cr-spec-converted-to-values-yaml>
3. Patch all of the new release resources AND the new release secret with an owner reference that references the CR.
4. Using the helm v3.2+ CLI, run helm uninstall <old-release-name-with-uid>
5. Start the v0.18+ operator

If you can tolerate downtime, you can also upgrade by doing the following:

1. With the v0.17 (or lower) operator running, delete all CRs to uninstall the release with the legacy name.
2. Rollout a new deployment of the operator to one running v0.18+
3. Re-create the CRs that were deleted in step 1.

[joelanford]

@esara and I chatted about his particular use case offline and it turns out that the above non-downtime procedure only works when all release resources contain the release name in their name. If release resources do NOT contain the release name (e.g. they're hardcoded), step 2 will fail with Helm complaining of existing resources.

A more general solution involves:

1. Stopping the operator.
2. Patching the release secret to remove all references to the UID portion of the release name.
3. Patching all release resources to remove the UID portion of the release name from the app.kubernetes.io/instance label value.
4. Starting the v0.18.0+ version of the operator and ensuring all CRs reconcile correctly. In this step, any resources that include the release name in their name will be created with the new name.
5. Deleting all of the orphaned resources from the old release (whose names include the UID portion of the release name).

[camilamacedo86]

Closing this one since shows sorted out by @joelanford. However, @esara feel free to ping us to re-open it if you see that it is required.

[esara]

to close the loop on this, we published a script that updates the secret that stores a uid based helm release to work properly with helm3/v0.18+ operator-sdk at https://github.com/turbonomic/t8c-install/blob/master/bin/updateAnnotations.sh#L132