ODH operator fails to patch PVCs probably due to changes storage class

[tumido]

Describe the bug
Seems like we have a mismatch between PVCs in cluster and manifests deployed via ODH operator:

secret/jupyterhub configured
configmap/odh-jupyterhub-sizes unchanged
configmap/spark-cluster-template unchanged
configmap/parameters unchanged
configmap/jupyter-singleuser-profiles unchanged
configmap/jupyterhub-cfg unchanged
configmap/jupyterhub-default-groups-config unchanged
serviceaccount/jupyterhub-hub unchanged
clusterrolebinding.rbac.authorization.k8s.io/jupyterhub-nodereader unchanged
role.authorization.openshift.io/jupyterhub configured
rolebinding.authorization.openshift.io/jupyterhub configured
service/jupyterhub-db unchanged
service/jupyterhub unchanged
deploymentconfig.apps.openshift.io/jupyterhub configured
deploymentconfig.apps.openshift.io/jupyterhub-db configured
imagestream.image.openshift.io/jupyterhub-img configured
route.route.openshift.io/jupyterhub configured
time="2021-06-03T15:23:43Z" level=warning msg="Encountered error applying application jupyterhub:  (kubeflow.error): Code 500 with message: Apply.Run : error when applying patch:\n{\"metadata\":{\"annotations\":{\"kubectl.kubernetes.io/last-applied-configuration\":\"{\\\"apiVersion\\\":\\\"v1\\\",\\\"kind\\\":\\\"PersistentVolumeClaim\\\",\\\"metadata\\\":{\\\"annotations\\\":{\\\"kfctl.kubeflow.io/kfdef-instance\\\":\\\"opendatahub.opf-jupyterhub\\\",\\\"volume.beta.kubernetes.io/storage-class\\\":\\\"moc-nfs-csi\\\"},\\\"labels\\\":{\\\"component.opendatahub.io/name\\\":\\\"jupyterhub\\\",\\\"opendatahub.io/component\\\":\\\"true\\\"},\\\"name\\\":\\\"jupyterhub-db\\\",\\\"namespace\\\":\\\"opf-jupyterhub\\\"},\\\"spec\\\":{\\\"accessModes\\\":[\\\"ReadWriteOnce\\\"],\\\"resources\\\":{\\\"requests\\\":{\\\"storage\\\":\\\"1Gi\\\"}}}}\\n\",\"volume.beta.kubernetes.io/storage-class\":\"moc-nfs-csi\"}}}\nto:\nResource: \"/v1, Resource=persistentvolumeclaims\", GroupVersionKind: \"/v1, Kind=PersistentVolume...
time="2021-06-03T15:23:43Z" level=warning msg="Will retry in 25 seconds."

It makes ODH operator stuck on a single KfDef and doesn't progress to a next one.

[4n4nd]

I think a temp fix would be removing the patch that specifies the storageclass

[HumairAK]

Okay so I'm proposing for now we revert back to ceph in the kfdef (the one on live is ceph anyway). Fix the reconcile errors.

Then we need to consider how to migrate to a new db using nfs backend.
I think we can just delete the current db. See here for what happens when you delete the db.

My concern is that I'm not sure how the culler gets informed on user pod idle'ness, so the next proposal is to deploy culler in stage to test this out before trying it on prod.

[sesheta]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[HumairAK]

Outdated issue from zero ODH deployment. Closing.