Che operator fails to reconcile che after failed installation of devworkspace

[metlos]

Describe the bug

I've had some issues during the installation of the che-operator with devworkspaces enabled and wanted to reinstall. I deleted eclipse-che and devworkspace-controller namespaces (by running kubectl delete namespace ... and deleting the finalizers on resources where necessary).

I then wanted to install che using chectl. It never succeeded with these errors in the che operator log:

time="2021-03-09T20:41:31Z" level=info msg="Running exec for 'create Keycloak DB, user, privileges' in the pod 'postgres-7d794f7b58-h2xkm'"
time="2021-03-09T20:41:31Z" level=error msg="Error running exec: Internal error occurred: failed calling webhook \"validate-exec.devworkspace-controller.svc\": Post \"https://devworkspace-webhookserver.devworkspace-controller.svc:443/validate?timeout=30s\": service \"devworkspace-webhookserver\" not found, command: [/bin/bash -c OUT=$(psql postgres -tAc \"SELECT 1 FROM pg_roles WHERE rolname='keycloak'\"); if [ $OUT -eq 1 ]; then echo \"DB exists\"; exit 0; fi && psql -c \"CREATE USER keycloak WITH PASSWORD 'is8NVHQ7r0GL'\" && psql -c \"CREATE DATABASE keycloak\" && psql -c \"GRANT ALL PRIVILEGES ON DATABASE keycloak TO keycloak\" && psql -c \"ALTER USER ${POSTGRESQL_USER} WITH SUPERUSER\"]"
time="2021-03-09T20:41:31Z" level=error msg="Stderr: "

Notice the reference to the devworkspace-webhookserver during the installation of the postgres DB for che server.

Because setting spec.devworkspace.enable: false actually does not uninstall devworkspace from the cluster in any way, the user has no way of making the installation work again.

Note that I was able to make the installation work again by running make uninstall from devworkspace-operator sources, but that is something that we might not want the users to do,

Che version

• latest

Steps to reproduce

1. Install Che using chectl server:deploy -p openshift -n eclipse-che -a operator
2. kubectl edit checluster eclipse-che -n eclipse-che and set spec.devworkspace.enable: true
3. Let the installation finish
4. kubectl delete namespace eclipse-che devworkspace-controller
5. Remove finalizers on resources blocking the deletion
6. Wait for the namespaces to be deleted
7. Try to install Che using chectl server:deploy -p openshift -n eclipse-che -a operator again

The installation never finishes with the error in the che-operator log as described above.

Expected behavior

We should have a documented way of cleaning up the cluster to be able to do repeated installations.

Runtime

• Openshift 4.6

Screenshots

N/A

Installation method

• see repro steps

Environment

• RHPDS with OpenShift 4.7

[amisevsk]

The uninstall process for the DevWorkspaceOperator is complicated and manual (e.g. the WTO docs or makefile uninstall rule).

Improper installation will break pods/exec across the cluster (kind of by design, but mostly to fill a gap in OLM, which doesn't allow us to force removal of all CRDs when an operator is uninstalled).

@sleshchenko I seem to recall we couldn't set an ObjectSelector on the validating webhook (hence causing this issue), but I can't find anything in the docs forbidding it. Could we just add a labelselector to our validating webhook to at least kind-of avoid this problem?

[tolusha]

Switching spec.devworkspace.enable: false should not uninstall dev workspace operator since it might be used by another Eclipse Che deployment.
To remove dev workspace operator you can try chectl server:delete command.
Besides removing Eclipse Che it removes devworkspace operator only if there are no another instance of Eclipse Che on the cluster.

[sleshchenko]

(by running kubectl delete namespace ... and deleting the finalizers on resources where necessary).

you went into very non-optimal way, in addition to deleting the finalizers, you just need to clean up all the Cluster-scoped resources, where webhooks are most critical.

Could we just add a labelselector to our validating webhook to at least kind-of avoid this problem?

pods labels are not propagated to pod/execs subresources, here is an issue on K8s side which should unblock us kubernetes/kubernetes#91732
Maybe it's changed but issue is still opened, but I doubt

[sleshchenko]

Switching spec.devworkspace.enable: false should not uninstall dev workspace operator since it might be used by another Eclipse Che deployment.

yes and no. If it does not uninstall it, it must provide instruction how to remove it properly.
I have doubt that in production Cluster is expected to have more than one Che/CRW installed.

If we care about development environments, I think Che Operator can store somewhere the info, which CheClusters uses DevWorkspace Operator, and the last can uninstall DWO and DevWorkspace Che Operator. It can be configmap in the DWO namespace.
Or CheCluster can get some testing field, which would allow to disable DWO uninstalling.

[amisevsk]

pods labels are not propagated to pod/execs subresources, here is an issue on K8s side which should unblock us

That was the roadblock I was remembering :)

[che-bot]

Issues go stale after 180 days of inactivity. lifecycle/stale issues rot after an additional 7 days of inactivity and eventually close.

Mark the issue as fresh with /remove-lifecycle stale in a new comment.

If this issue is safe to close now please do so.

Moderators: Add lifecycle/frozen label to avoid stale mode.

[mmorhun]

Still actual. I see the same error which prevents me now from debugging Che operator

[tolusha]

We can use now chectl server:delete to remove Eclipse Che + clean up DevWorkspace resources.