Upgrading Rancher from 2.4.8 -> 2.5.0 fails with Cert Manager v1.0.1 #29213

bmdepesa · 2020-09-28T17:00:22Z

rancher/rancher:v2.4.8 -> v2.5.0-rc2

Install Rancher w/ cert manager v1.0.1 utilizing LetsEncrypt certs (from Jenkins automation)
Create some clusters
Attempt to upgrade Rancher with helm upgrade
Upgrade fails with error:

Unable to continue with update: Issuer "rancher" in namespace "cattle-system" exists and cannot be imported into the current release: invalid ownership metadata; label validation error: missing key "app.kubernetes.io/managed-by": must be set to "Helm"; annotation validation error: missing key "meta.helm.sh/release-name": must be set to "rancher"; annotation validation error: missing key "meta.helm.sh/release-namespace": must be set to "cattle-system"

This does not occur when using cert manager v0.15.0

Question
Should the fresh install of v2.4.8 work with cert manager v1.0.1?

The text was updated successfully, but these errors were encountered:

aaronyeeski · 2020-09-28T17:08:26Z

The same steps produce a slightly different error when using Rancher self signed certs. Local k8s version v1.19.2.
RKE version v1.2.0-rc15
Helm Version:"v3.0.1
Cert manager command:

helm_v3 install cert-manager jetstack/cert-manager --namespace cert-manager --version v0.15.0

Rancher install command:

helm_v3 install rancher rancher-latest/rancher --version 2.4.8 --namespace ****-system --set hostname=***

Upgrade command:

helm upgrade rancher rancher-latest/rancher --version 2.5.0-rc2 --namespace cattle-system --set hostname=***

Error:

Error: UPGRADE FAILED: rendered manifests contain a new resource that already exists. Unable to continue with update: existing resource conflict: kind: Issuer, namespace: cattle-system, name: rancher

rawmind0 · 2020-09-28T17:59:20Z

I've tried to reproduce the issue but installation and upgrade are working fine for me. I'm testing with k8s v1.18.9 and helm v3.3.3 What k8s version and helm client version are you using??

I'm guessing that the issue may be related to cert-manager crd issue, related with k8s issue. Could you please try using using k8s v1.18.9 and helm v3.3.3??

bmdepesa · 2020-09-28T23:40:08Z

Attempted with:

Fresh install with helm v3.0.2
Helm v3.3.4
v1.18.9+k3s1

To the same result

rawmind0 · 2020-09-28T23:56:08Z

Could you please paste you helm command for install and upgrade??

Both working fine for me using these:

cert-manager v1.0.1, helm install cert-manager jetstack/cert-manager --namespace cert-manager --version v1.0.1
Rancher 2.4.8 install (self signed), helm install rancher rancher-latest/rancher --namespace cattle-system --set hostname=rancher.172.17.0.2.xip.io --version 2.4.8
rancher 2.5.0-rc2 upgrade (self signed), helm upgrade rancher rancher-latest/rancher --namespace cattle-system --set hostname=rancher.172.17.0.2.xip.io --version 2.5.0-rc2
rancher 2.4.8 install (letsencrypt) helm install rancher rancher-latest/rancher --namespace cattle-system --set hostname=rancher.172.17.0.2.xip.io --set ingress.tls.source=letsEncrypt --set letsEncrypt.email=raul@rancher.com --version 2.4.8
rancher 2.5.0-rc2 upgrade (letsencrypt) helm upgrade rancher rancher-latest/rancher --namespace cattle-system --set hostname=rancher.172.17.0.2.xip.io --set ingress.tls.source=letsEncrypt --set letsEncrypt.email=raul@rancher.com --version 2.5.0-rc2

bmdepesa · 2020-09-29T18:09:05Z

The following is successful

k8s 1.18.8 (rke v1.1.7)
helm 3.3.4
kubectl 1.19.2

rke up

kubectl create ns cert-manager

kubectl create ns cattle-system

kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.0.1/cert-manager.crds.yaml

helm install cert-manager jetstack/cert-manager --namespace cert-manager --version v1.0.1

helm install rancher rancher-latest/rancher --version 2.4.8 --namespace cattle-system --set hostname=host --set ingress.tls.source=letsEncrypt --set letsEncrypt.email=email

helm upgrade rancher rancher-latest/rancher --version 2.5.0-rc2 --namespace cattle-system --set hostname=host --set ingress.tls.source=letsEncrypt --set letsEncrypt.email=email

So far it appears that we reliably see the issue when we use a lower version of Helm (v3.0.2), and then try to upgrade with a higher version (3.3.4).

rawmind0 · 2020-09-29T18:51:07Z

On our tests, we've noticed that the error is related with the helm version. It's occurring when helm 3.0.x or 3.1.x is used to install, then any helm version is failing to upgrade. Using helm 3.2.x or above instead, is working fine on install and upgrade.

maggieliu · 2020-09-29T20:59:53Z

We will document that user needs 3.2.x + to install/upgrade Rancher.

rawmind0 · 2021-01-13T12:18:53Z

Helm version requirements is already documented, https://rancher.com/docs/rancher/v2.x/en/installation/resources/helm-version/

jaimehrubiks · 2021-01-19T16:10:03Z

I still cannot update my rancher after updating helm (tried helm 3.2, 3.3, 3.4...) This uses self-signed certs, and the values.yaml is

hostname: rancher.nix.example.com
ingress:
  tls:
    source: secret

I don't have certs manager installed, never needed it

[rancher-provisioner@dg-jumphost-01 ~]$ helm upgrade rancher rancher-latest/rancher --namespace cattle-system -f values19jan2021.yaml --version=2.5.5
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /home/rancher-provisioner/.kube/config
Error: UPGRADE FAILED: rendered manifests contain a resource that already exists. Unable to continue with update: Ingress "rancher" in namespace "cattle-system" exists and cannot be imported into the current release: invalid ownership metadata; label validation error: missing key "app.kubernetes.io/managed-by": must be set to "Helm"; annotation validation error: missing key "meta.helm.sh/release-name": must be set to "rancher"; annotation validation error: missing key "meta.helm.sh/release-namespace": must be set to "cattle-system"

[rancher-provisioner@dg-jumphost-01 ~]$ helm version
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /home/rancher-provisioner/.kube/config
version.BuildInfo{Version:"v3.4.0", GitCommit:"7090a89efc8a18f3d8178bf47d2462450349a004", GitTreeState:"clean", GoVersion:"go1.14.10"}

@rawmind0 was helm version the only issue that you had on your tests?

rawmind0 · 2021-01-19T16:57:05Z

@rawmind0 was helm version the only issue that you had on your tests?

Yes, that was the only issue i had. Have you tested this #29213 (comment) ?? Are you installing and upgrading with same helm version??

Anyway, your issue seems a little bit different. This was related to the use of cert-manager not self-signed certs.

santachago · 2021-01-19T18:35:52Z

Hi i seem to be having the same issue as yours updating, but in my case i dont have Cert manager as i am running with External TLS Termination.
Is Cert Manager a requirement from version 2.5?

jaimehrubiks · 2021-01-19T20:46:22Z

I solved my issue with an idea I got from @rawmind0 comment

Basically I installed rancher 2.4.5 with helm 3.1, so what I did is:

upgrade helm to 3.2
upgrade rancher to same version with helm 3.2 (helm upgrade --version 2.4.5)
upgrade rancher to next version with helm 3.2 (helm upgrade --version 2.5.5)

(Note that after that I had some issue which could be unrelated to this issue that made me have to restore my RKE cluster from etcd backup, so regardless of your situation always backup your rancher/cluster or better yet, your VMs)

Compukid · 2021-01-22T06:03:25Z

Sorry, I am new to rancher and following someone from youtube, but the upgrade video is a little old and it seems like you updated a few thing since then. Currently running Rancher v2.4.8 on a single node (I know it is not correct, but this is just my home install) How does one upgrade helm? the part that is confusing me is helm. Do I use a help command to upgrade rancher or I just need helm 3.2 and then I can upgrade rancher to v 2.5.5 ? Any help / guidance is very welcome.

mshade · 2021-02-17T22:54:10Z

We're using external TLS termination with a purchased certificate. We upgraded helm from 2.x to 3.x using helm 2to3 plugin. After that, helm 3.5 was failing to upgrade from Rancher 2.4.13 to 2.5.5 (with --dry-run) with the following:

Error: UPGRADE FAILED: rendered manifests contain a resource that already exists.
Unable to continue with update: Ingress "rancher" in namespace "cattle-system" exists and cannot be imported into the current release: invalid ownership metadata; label validation error: missing key "app.kubernetes.io/managed-by": must be set to "Helm";
 annotation validation error: missing key "meta.helm.sh/release-name": must be set to "rancher";
 annotation validation error: missing key "meta.helm.sh/release-namespace": must be set to "cattle-system"

To fix:

redeploy 2.4.13 with existing values and helm3. This applies the missing labels.
change version to 2.5.5, redeploy with helm3. Deploy succeeds.

bmdepesa added area/server-chart kind/bug-qa Issues that have not yet hit a real release. Bugs introduced by a new feature or enhancement labels Sep 28, 2020

bmdepesa added this to the v2.5 milestone Sep 28, 2020

bmdepesa assigned rawmind0, bmdepesa and aaronyeeski Sep 28, 2020

maggieliu added the [zube]: Next Up label Sep 28, 2020

maggieliu added the area/documentation label Sep 29, 2020

maggieliu added [zube]: Team Blue Backlog and removed [zube]: Next Up labels Sep 29, 2020

maggieliu modified the milestones: v2.5, v2.5.x Sep 30, 2020

mrajashree mentioned this issue Sep 30, 2020

Restoring in a new rancher setup fails rancher/backup-restore-operator#44

Closed

maggieliu modified the milestones: v2.5.x, v2.5 - Backlog Oct 2, 2020

rawmind0 closed this as completed Jan 13, 2021

rawmind0 added [zube]: Done and removed [zube]: Team Blue Backlog labels Jan 13, 2021

deniseschannon modified the milestones: v2.5 - Backlog, v2.x Issues Closed without a specific milestone Jan 31, 2021

zube bot removed the [zube]: Done label Apr 14, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrading Rancher from 2.4.8 -> 2.5.0 fails with Cert Manager v1.0.1 #29213

Upgrading Rancher from 2.4.8 -> 2.5.0 fails with Cert Manager v1.0.1 #29213

bmdepesa commented Sep 28, 2020

aaronyeeski commented Sep 28, 2020 •

edited

Loading

rawmind0 commented Sep 28, 2020 •

edited

Loading

bmdepesa commented Sep 28, 2020 •

edited

Loading

rawmind0 commented Sep 28, 2020 •

edited

Loading

bmdepesa commented Sep 29, 2020 •

edited

Loading

rawmind0 commented Sep 29, 2020

maggieliu commented Sep 29, 2020

rawmind0 commented Jan 13, 2021

jaimehrubiks commented Jan 19, 2021 •

edited

Loading

rawmind0 commented Jan 19, 2021

santachago commented Jan 19, 2021

jaimehrubiks commented Jan 19, 2021 •

edited

Loading

Compukid commented Jan 22, 2021

mshade commented Feb 17, 2021 •

edited

Loading

Upgrading Rancher from 2.4.8 -> 2.5.0 fails with Cert Manager v1.0.1 #29213

Upgrading Rancher from 2.4.8 -> 2.5.0 fails with Cert Manager v1.0.1 #29213

Comments

bmdepesa commented Sep 28, 2020

aaronyeeski commented Sep 28, 2020 • edited Loading

rawmind0 commented Sep 28, 2020 • edited Loading

bmdepesa commented Sep 28, 2020 • edited Loading

rawmind0 commented Sep 28, 2020 • edited Loading

bmdepesa commented Sep 29, 2020 • edited Loading

rawmind0 commented Sep 29, 2020

maggieliu commented Sep 29, 2020

rawmind0 commented Jan 13, 2021

jaimehrubiks commented Jan 19, 2021 • edited Loading

rawmind0 commented Jan 19, 2021

santachago commented Jan 19, 2021

jaimehrubiks commented Jan 19, 2021 • edited Loading

Compukid commented Jan 22, 2021

mshade commented Feb 17, 2021 • edited Loading

aaronyeeski commented Sep 28, 2020 •

edited

Loading

rawmind0 commented Sep 28, 2020 •

edited

Loading

bmdepesa commented Sep 28, 2020 •

edited

Loading

rawmind0 commented Sep 28, 2020 •

edited

Loading

bmdepesa commented Sep 29, 2020 •

edited

Loading

jaimehrubiks commented Jan 19, 2021 •

edited

Loading

jaimehrubiks commented Jan 19, 2021 •

edited

Loading

mshade commented Feb 17, 2021 •

edited

Loading