upgrade failure

[13567436138]

Description

root@karmada-01-b:~/openebs# helm upgrade openebs .  -n openebs  --kubeconfig ~/.kube/135config 
Error: UPGRADE FAILED: cannot patch "volumesnapshotclasses.snapshot.storage.k8s.io" with kind CustomResourceDefinition: customresourcedefinitions.apiextensions.k8s.io "volumesnapshotclasses.snapshot.storage.k8s.io" is invalid: metadata.resourceVersion: Invalid value: 0x0: must be specified for an update && cannot patch "volumesnapshotcontents.snapshot.storage.k8s.io" with kind CustomResourceDefinition: customresourcedefinitions.apiextensions.k8s.io "volumesnapshotcontents.snapshot.storage.k8s.io" is invalid: metadata.resourceVersion: Invalid value: 0x0: must be specified for an update && cannot patch "volumesnapshots.snapshot.storage.k8s.io" with kind CustomResourceDefinition: customresourcedefinitions.apiextensions.k8s.io "volumesnapshots.snapshot.storage.k8s.io" is invalid: metadata.resourceVersion: Invalid value: 0x0: must be specified for an update && cannot patch "openebs-localpv-provisioner" with kind Deployment: Deployment.apps "openebs-localpv-provisioner" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app":"localpv-provisioner", "component":"localpv-provisioner", "release":"openebs"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable

Expected Behavior

Current Behavior

Possible Solution

Steps to Reproduce

Screenshots

Your Environment

• kubectl get nodes:
• kubectl get pods --all-namespaces:
• kubectl get services:
• kubectl get sc:
• kubectl get pv:
• kubectl get pvc:
• OS (from /etc/os-release):
• Kernel (from uname -a):
• Install tools:
• Others:

[sjerman]

I've got this as well.

Upgrade
microk8s 1.26 -> 1.28

Then
openebs v3.7.1 -> 3.10.0

get the error above.

[devrushit]

having the same issue 👍🏼

[tiagolobocastro]

We have are moving the CR's to a separate chart dependency which can be disabled (will be released on next version).
@niladrih anything else to add here?

[devrushit]

Nice thanks for your reply, when is the release planned ?

[tiagolobocastro]

Sometime in April :)

[devrushit]

used a dirty workaround by adding the missing labels to run it over. Really dirty, but works. If someone is having the same issue.

[orville-wright]

I believe this issue was fixed with the v4.0.0 and v4.0.1 releases.
@tiagolobocastro @niladrih @avishnu Please confirm? - and we can CLOSE this issue.

[niladrih]

@13567436138 -- hi, could try this upgrade user-guide here: https://openebs.io/docs/user-guides/upgrade

[antonblr]

I'm getting the following error trying to deploy 4.0.1 over 3.10 (via Flux CD Helm definitions)

openebs              	openebs              	3.10.0  	False    	False	

Helm upgrade failed: rendered manifests contain a resource that already exists. 
Unable to continue with update: CustomResourceDefinition "volumesnapshotclasses.snapshot.storage.k8s.io" 
in namespace "" exists and cannot be imported into the current release:
invalid ownership metadata; label validation error: missing key "app.kubernetes.io/managed-by": 
must be set to "Helm"; annotation validation error: missing key "meta.helm.sh/release-name": must be set to "openebs"; annotation validation error: missing key "meta.helm.sh/release-namespace": 
must be set to "openebs"

EDIT: current annotations and labels:

metadata:
  annotations:
    api-approved.kubernetes.io: https://github.com/kubernetes-csi/external-snapshotter/pull/814
    controller-gen.kubebuilder.io/version: v0.11.3
    helm.sh/resource-policy: keep
  labels:
    helm.toolkit.fluxcd.io/name: openebs
    helm.toolkit.fluxcd.io/namespace: openebs

But I don't see that resource in the cluster at all: nvm, it is crd

$ kubectl get volumesnapshotclasses.snapshot.storage.k8s.io -A
No resources found

Chart values for 3.10 (https://openebs.github.io/charts) - all defaults. Which installs openebs-ndm and localpv-provisioner only (ref: https://github.com/openebs-archive/charts/tree/main/charts/openebs#how-to-customize-openebs-helm-chart)

Chart values for 4.0.1 (disabled all, as we're interested in localpv-provisioner only):

engines:
  local:
    lvm:
      enabled: false
    zfs:
      enabled: false
  replicated:
    mayastor:
      enabled: false

UPDATE

Disabling openebs-crds.csi.volumeSnapshots seem to solve the issue. I see some bits about disabling that in https://openebs.io/docs/user-guides/upgrade#replicated-storage, but it is under "Replicated Storage" section, which I skipped, since it does not apply to our case.

openebs-crds:
  csi:
    volumeSnapshots:
      enabled: false