Workspace start failed on OCP because it was unable to perform mkdir command

[dmytro-ndp]

Describe the bug

Eclipse Che 7.14.1 workspace failed to start because of the next error:

2020-06-05 21:32:23,755[nio-8080-exec-2] [INFO ] [o.e.c.a.w.s.WorkspaceManager 651] - Workspace 'admin/java-maven-kgvam' with id 'workspaceb06etedf8yvi3lb9' created by user 'admin'
2020-06-05 21:32:26,540[nio-8080-exec-3] [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 479] - Starting workspace 'admin/java-maven-kgvam' with id 'workspaceb06etedf8yvi3lb9' by user 'admin'

2020-06-05 21:32:27,407[aceSharedPool-0] [ERROR] [e.c.w.i.k.n.p.PVCSubPathHelper 223] - Unable to perform '[mkdir, -m, 777, -p, /tmp/job_mount/workspaceb06etedf8yvi3lb9/che-logs-che-plugin-broker/]' command for the workspace 'workspaceb06etedf8yvi3lb9' cause: 'Failure executing: POST at: https://172.30.0.1/api/v1/namespaces/che-chectl-tls-no-oauth/pods. Message: admission webhook "mutate-ws-resources.che-workspace-controller.svc" denied the request: 'org.eclipse.che.workspace/creator' label is missing. Received status: Status(apiVersion=v1, code=400, details=null, kind=Status, message=admission webhook "mutate-ws-resources.che-workspace-controller.svc" denied the request: 'org.eclipse.che.workspace/creator' label is missing, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=null, status=Failure, additionalProperties={}).'

An update: there was a similar error on OCP 4.4 when tried to start workspace on Eclipse Che 7.14.1 with TLS and OS OAuth support:

2020-06-09 17:04:44,497[ceSharedPool-16] [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 944] - Workspace 'admin:angular-ban8f' with id 'workspacev9a1z4moooovj01p' start failed
2020-06-09 17:30:47,953[nio-8080-exec-8] [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 479] - Starting workspace 'admin/angular-ban8f' with id 'workspacev9a1z4moooovj01p' by user 'admin'
2020-06-09 17:35:54,474[ceSharedPool-17] [ERROR] [e.c.w.i.k.n.p.PVCSubPathHelper 223] - Unable to perform '[mkdir, -m, 777, -p, /tmp/job_mount/workspacev9a1z4moooovj01p/che-logs-che-plugin-broker/]' command for the workspace 'workspacev9a1z4moooovj01p' cause: 'Waiting for pod 'mkdir-workspacev9a1z4moooovj01p' reached timeout'

Che version

• latest
• nightly
• other: Eclipse Che 7.14.1

Steps to reproduce

1. Go to Eclipse Che 7.14.1 instance on QE OCP.
2. Try to create workspace of any type from Dashboard.

Expected behavior

Workspace successfully started

Runtime

• kubernetes (include output of kubectl version)
• OCP 4.5.0-0.nightly-2020-06-03-045340, OCP 4.4.
• minikube (include output of minikube version and kubectl version)
• minishift (include output of minishift version and oc version)
• docker-desktop + K8S (include output of docker version and kubectl version)
• other: (please specify)

Screenshots

Installation method

• chectl 7.14.1
  command to install:

chectl/bin/chectl server:start --listr-renderer=verbose --debug --multiuser --installer=operator --platform=openshift -n che-chectl-tls-no-oauth --che-operator-cr-patch-yaml=/mnt/hudson_workspace/workspace/install-che-using-chectl/custom-resource-patch.yaml

custom-resource-patch.yaml:

spec:
  server:
    selfSignedCert: true
    customCheProperties:
      CHE_LIMITS_USER_WORKSPACES_RUN_COUNT: '-1'
      CHE_WORKSPACE_AGENT_DEV_INACTIVE__STOP__TIMEOUT__MS: '300000'
  auth:
    updateAdminPassword: false
    identityProviderPassword: admin

• OperatorHub
• I don't know

Environment

• my computer
  • Windows
  • Linux
  • macOS
• Cloud
  • Amazon
  • Azure
  • GCE
  • other (please specify)
• other: CRW CCI

Eclipse Che Logs

https://codeready-workspaces-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/java-tests/796/artifact/logs-and-configs/logs/che-548cd94f8d-cfbq5.pod.log

Additional context

[skabashnyuk]

Probably related to this code https://github.com/che-incubator/che-workspace-operator/blob/master/pkg/webhook/workspace/mutating_cfg.go#L60

[skabashnyuk]

Expected behavior #16924

[skabashnyuk]

@sleshchenko To workaround it please remove mutating and validating workspace related webhhoks. Or just uninstall controller - 'make uninstall' from controller project folder

[nickboldt]

Will #16924 be fixed in time for 7.14.2?

[sleshchenko]

@dmytro-ndp I believe that the added similar issue if just similar but not the same?
Could you separate these two issues?
BTW the issue with webhooks will disappear tomorrow after devfile/devworkspace-operator#99 is merged and nightly is rebuilt

[sleshchenko]

Should be finally fixed by devfile/devworkspace-operator#99