helm 3 beta 5, cannot uninstall releases done by same ver

[masterkain]

there are always release secrets that cannot be removed automatically.

helm3 uninstall nginx-ingress -n ingress-nginx
Error: uninstallation completed with 1 error(s): uninstall: Failed to purge the release: delete: failed to get release "sh.helm.release.v1.nginx-ingress.v1": release: not found

[karuppiah7890]

@masterkain Sorry about the issue. The issue here is, there was a breaking change in between the beta releases, where we changed the way releases are stored - the names of the secrets / configmaps basically. This breaking change happened in bet4. So, if you used v3-beta3 or older versions to install, then you will have to use those to delete the releases.

Here's an example of what I'm telling

$ helm3-beta3 install test --generate-name
NAME: test-1572068781
LAST DEPLOYED: 2019-10-26 11:16:21.782769 +0530 IST m=+0.098977586
NAMESPACE: default
STATUS: deployed
NOTES:
1. Get the application URL by running these commands:
  export POD_NAME=$(kubectl get pods -l "app=test,release=test-1572068781" -o jsonpath="{.items[0].metadata.name}")
  echo "Visit http://127.0.0.1:8080 to use your application"
  kubectl port-forward $POD_NAME 8080:80
$ k get secrets
NAME                  TYPE                                  DATA   AGE
default-token-dzgwv   kubernetes.io/service-account-token   3      71d
draft-pullsecret      kubernetes.io/dockercfg               1      68d
test-1572068781.v1    helm.sh/release                       1      5s
$ helm3-beta4 list
NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART     A
PP VERSION
test-1572068781 default         1               2019-10-26 11:16:21.782769 +0530 IST    deployed        test-0.1.01
.16.0
$ helm3-beta4 delete test
Error: uninstall: Release not loaded: test: release: not found
$ helm3-beta4 delete test-1572068781
Error: uninstallation completed with 1 error(s): uninstall: Failed to purge the release: delete: failed to get rele
ase "sh.helm.release.v1.test-1572068781.v1": release: not found
$ helm3-beta3 delete test-1572068781
release "test-1572068781" uninstalled

When installing with helm3-beta4, this is how secrets will look:

$ helm3-beta4 install test test
NAME: test
LAST DEPLOYED: Sat Oct 26 11:20:11 2019
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:
1. Get the application URL by running these commands:
  export POD_NAME=$(kubectl get pods -l "app=test,release=test" -o jsonpath="{.items[0].metadata.name}")
  echo "Visit http://127.0.0.1:8080 to use your application"
  kubectl port-forward $POD_NAME 8080:80
$ k get secrets
NAME                         TYPE                                  DATA   AGE
default-token-dzgwv          kubernetes.io/service-account-token   3      71d
draft-pullsecret             kubernetes.io/dockercfg               1      68d
sh.helm.release.v1.test.v1   helm.sh/release.v1                    1      5s
$ helm3-beta3 list
NAME    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART
test    default         1               2019-10-26 11:20:11.80007 +0530 IST     deployed        test-0.1.0
$ helm3-beta3 delete test
Error: uninstallation completed with 1 error(s): uninstall: Failed to purge the release: delete: failed to get rele
ase "test.v1": release: not found
$ helm3-beta4 delete test
release "test" uninstalled

As you can see helm3-beta4 (or newer) releases cannot be deleted by older versions

[karuppiah7890]

Here's the PR that made the change #6500 and the reason was this issue - #6435

[masterkain]

hello @karuppiah7890 thanks for the support -- I'm aware of the other issues, what I perhaps did not mention is that I always used beta 5 to do all my setup

[karuppiah7890]

Ah. That's weird. Could you list the secrets you have using kubectl get secrets -n ingress-nginx ?

[masterkain]

kain@kain-osx-qemu ~ % kubectl get secrets
NAME                             TYPE                                  DATA   AGE
default-token-svcdz              kubernetes.io/service-account-token   3      46h
sh.helm.release.v1.linkerd2.v1   helm.sh/release.v1                    1      46h


kain@kain-osx-qemu ~ % kubectl get secrets --all-namespaces
NAMESPACE              NAME                                             TYPE                                  DATA   AGE
default                default-token-svcdz                              kubernetes.io/service-account-token   3      46h
default                sh.helm.release.v1.linkerd2.v1                   helm.sh/release.v1                    1      46h
elastic-system         default-token-cb5jz                              kubernetes.io/service-account-token   3      3m58s
elastic-system         elastic-operator-token-grtjf                     kubernetes.io/service-account-token   3      3m58s
errbit                 default-token-2p4bt                              kubernetes.io/service-account-token   3      8h
errbit                 errbit-errbit-helm                               Opaque                                2      8h
errbit                 errbit-mongodb                                   Opaque                                2      8h
errbit                 errbit.v1                                        helm.sh/release                       1      8h
fluxcd                 default-token-qt84q                              kubernetes.io/service-account-token   3      46h
fluxcd                 flux-git-deploy                                  Opaque                                1      46h
fluxcd                 flux-helm-repositories                           Opaque                                1      46h
fluxcd                 flux-token-vg99v                                 kubernetes.io/service-account-token   3      46h
fluxcd                 helm-operator-token-bc66x                        kubernetes.io/service-account-token   3      46h
fluxcd                 sh.helm.release.v1.flux.v1                       helm.sh/release.v1                    1      46h
fluxcd                 sh.helm.release.v1.flux.v2                       helm.sh/release.v1                    1      40h
fluxcd                 sh.helm.release.v1.helm-operator.v1              helm.sh/release.v1                    1      46h
ingress-nginx          aws-alb-ingress-controller-token-dx79w           kubernetes.io/service-account-token   3      40h
ingress-nginx          aws-alb-ingress-controller.v1                    helm.sh/release                       1      40h
ingress-nginx          default-token-w2ks2                              kubernetes.io/service-account-token   3      46h
ingress-nginx          nginx-ingress-backend-token-b4pwn                kubernetes.io/service-account-token   3      43h
ingress-nginx          nginx-ingress-token-vqkq7                        kubernetes.io/service-account-token   3      8h
ingress-nginx          nginx-ingress.v1                                 helm.sh/release                       1      8h
kube-node-lease        default-token-lf7hk                              kubernetes.io/service-account-token   3      46h
kube-public            default-token-6vnjm                              kubernetes.io/service-account-token   3      46h
kube-system            attachdetach-controller-token-2955c              kubernetes.io/service-account-token   3      46h
kube-system            aws-cloud-provider-token-4b5xg                   kubernetes.io/service-account-token   3      46h
kube-system            aws-node-token-bvsgq                             kubernetes.io/service-account-token   3      46h
kube-system            certificate-controller-token-lhcml               kubernetes.io/service-account-token   3      46h
kube-system            clusterrole-aggregation-controller-token-qgdx9   kubernetes.io/service-account-token   3      46h
kube-system            coredns-token-jtrdn                              kubernetes.io/service-account-token   3      46h
kube-system            cronjob-controller-token-4zp2h                   kubernetes.io/service-account-token   3      46h
kube-system            daemon-set-controller-token-nkfhh                kubernetes.io/service-account-token   3      46h
kube-system            default-token-trfz4                              kubernetes.io/service-account-token   3      46h
kube-system            deployment-controller-token-4mt7f                kubernetes.io/service-account-token   3      46h
kube-system            disruption-controller-token-d5q9v                kubernetes.io/service-account-token   3      46h
kube-system            eks-admin-token-2r5tp                            kubernetes.io/service-account-token   3      46h
kube-system            endpoint-controller-token-7n4c7                  kubernetes.io/service-account-token   3      46h
kube-system            expand-controller-token-mzjgq                    kubernetes.io/service-account-token   3      46h
kube-system            generic-garbage-collector-token-wqrhc            kubernetes.io/service-account-token   3      46h
kube-system            horizontal-pod-autoscaler-token-m27s9            kubernetes.io/service-account-token   3      46h
kube-system            job-controller-token-89czw                       kubernetes.io/service-account-token   3      46h
kube-system            kube-proxy-token-8tsxd                           kubernetes.io/service-account-token   3      46h
kube-system            metrics-server-token-km4nq                       kubernetes.io/service-account-token   3      46h
kube-system            namespace-controller-token-2mgl7                 kubernetes.io/service-account-token   3      46h
kube-system            node-controller-token-9fwrh                      kubernetes.io/service-account-token   3      46h
kube-system            persistent-volume-binder-token-f7npz             kubernetes.io/service-account-token   3      46h
kube-system            pod-garbage-collector-token-c8fwc                kubernetes.io/service-account-token   3      46h
kube-system            pv-protection-controller-token-bdpn2             kubernetes.io/service-account-token   3      46h
kube-system            pvc-protection-controller-token-ftptc            kubernetes.io/service-account-token   3      46h
kube-system            replicaset-controller-token-pxfgn                kubernetes.io/service-account-token   3      46h
kube-system            replication-controller-token-9nhcf               kubernetes.io/service-account-token   3      46h
kube-system            resourcequota-controller-token-kddxt             kubernetes.io/service-account-token   3      46h
kube-system            service-account-controller-token-r48nl           kubernetes.io/service-account-token   3      46h
kube-system            service-controller-token-zgzgz                   kubernetes.io/service-account-token   3      46h
kube-system            statefulset-controller-token-xvdqn               kubernetes.io/service-account-token   3      46h
kube-system            ttl-controller-token-m682q                       kubernetes.io/service-account-token   3      46h
kubernetes-dashboard   default-token-s7x55                              kubernetes.io/service-account-token   3      46h
kubernetes-dashboard   kubernetes-dashboard-certs                       Opaque                                0      46h
kubernetes-dashboard   kubernetes-dashboard-csrf                        Opaque                                1      46h
kubernetes-dashboard   kubernetes-dashboard-key-holder                  Opaque                                2      46h
kubernetes-dashboard   kubernetes-dashboard-token-sp8g6                 kubernetes.io/service-account-token   3      46h
linkerd                default-token-484r8                              kubernetes.io/service-account-token   3      46h
linkerd                flagger-token-fsm6w                              kubernetes.io/service-account-token   3      46h
linkerd                linkerd-controller-token-mhktd                   kubernetes.io/service-account-token   3      46h
linkerd                linkerd-destination-token-b8hb5                  kubernetes.io/service-account-token   3      46h
linkerd                linkerd-grafana-token-f2zxd                      kubernetes.io/service-account-token   3      46h
linkerd                linkerd-heartbeat-token-mcn26                    kubernetes.io/service-account-token   3      46h
linkerd                linkerd-identity-issuer                          Opaque                                2      46h
linkerd                linkerd-identity-token-d5q2s                     kubernetes.io/service-account-token   3      46h
linkerd                linkerd-prometheus-token-lk9hh                   kubernetes.io/service-account-token   3      46h
linkerd                linkerd-proxy-injector-tls                       Opaque                                2      46h
linkerd                linkerd-proxy-injector-token-c6m7s               kubernetes.io/service-account-token   3      46h
linkerd                linkerd-sp-validator-tls                         Opaque                                2      46h
linkerd                linkerd-sp-validator-token-hktpl                 kubernetes.io/service-account-token   3      46h
linkerd                linkerd-tap-tls                                  Opaque                                2      46h
linkerd                linkerd-tap-token-rd9tf                          kubernetes.io/service-account-token   3      46h
linkerd                linkerd-web-token-dxgsh                          kubernetes.io/service-account-token   3      46h
linkerd                sh.helm.release.v1.flagger.v1                    helm.sh/release.v1                    1      46h
prod                   bstore-prod-rails-env                            Opaque                                3      8h
prod                   bstore-prod.v1                                   helm.sh/release                       1      8h
prod                   default-token-rws7g                              kubernetes.io/service-account-token   3      45h
prod                   podinfo.v1                                       helm.sh/release                       1      45h
prod                   podinfo.v2                                       helm.sh/release                       1      43h
prod                   podinfo.v3                                       helm.sh/release                       1      40h
prod                   podinfo.v4                                       helm.sh/release                       1      40h
prod                   podinfo.v5                                       helm.sh/release                       1      40h

[karuppiah7890]

Looking at this

ingress-nginx          nginx-ingress.v1                                 helm.sh/release   

it seems to me like an older beta version was used to install the release. As the name would be different if it was helm3-beta5

[karuppiah7890]

Could you use helm3-beta3 and delete the release and try again with helm3-beta5 and let us know?

[masterkain]

it happens on every HelmRelease, most stuff I'm deploying with Helm Operator..

oh.. maybe that's the reason:

note the --set image.tag=helm-v3-71bc9d62

noglob helm3 upgrade -i helm-operator fluxcd/helm-operator --wait \
--namespace fluxcd \
--set git.ssh.secretName=flux-git-deploy \
--set git.pollInterval=1m \
--set chartsSyncInterval=1m \
--set configureRepositories.enable=true \
--set configureRepositories.repositories[0].name=stable \
--set configureRepositories.repositories[0].url=https://kubernetes-charts.storage.googleapis.com \
--set extraEnvs[0].name=HELM_VERSION \
--set extraEnvs[0].value=v3 \
--set image.repository=docker.io/fluxcd/helm-operator-prerelease \
--set image.tag=helm-v3-71bc9d62

[karuppiah7890]

Yes, that version uses helm3-beta3 - https://github.com/fluxcd/helm-operator/blob/71bc9d62c9654a7ea568dd37dda8f8a7bdbd4667/go.mod#L29

[karuppiah7890]

@masterkain I'll raise a PR to their repo. Or you would like to do it? 😄

[masterkain]

you're certainly most indicated, thanks a bunch!

[karuppiah7890]

Thanks :) and you want to raise the PR? Or I'll do it :)

[masterkain]

yeah thanks, go ahead, I meant that undoubtedly you're more qualified to raise such PR, cheers

[karuppiah7890]

Oh no. Not really. Now I want you to do it :p