[Hosted Che] permission denied while creating volume path during workspace startup against 7.9.0-SNAPSHOT version

[ibuziuk]

Describe the bug

While testing 7.9.0-SNAPSHOT against dev-cluster the following issue was discovered during any workspace startup:

pulling image "quay.io/eclipse/che-plugin-metadata-broker:v3.1.1"
Successfully pulled image "quay.io/eclipse/che-plugin-metadata-broker:v3.1.1"
Created container
Started container
Starting plugin metadata broker
All plugin metadata has been successfully processed
List of plugins and editors to install

• redhat/java11/latest - Java Linting, Intellisense, formatting, refactoring, Maven/Gradle support and more...
• eclipse/che-machine-exec-plugin/nightly - Che Plug-in with che-machine-exec service to provide creation terminal or tasks for Eclipse CHE workspace containers.
• eclipse/che-theia/next - Eclipse Theia, get the latest release each day.
  pulling image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next"
  Successfully pulled image "quay.io/eclipse/che-theia-endpoint-runtime-binary:next"
  Created container
  Started container
  pulling image "quay.io/eclipse/che-plugin-artifacts-broker:v3.1.1"
  Successfully pulled image "quay.io/eclipse/che-plugin-artifacts-broker:v3.1.1"
  Created container
  Started container
  Starting plugin artifacts broker
  Cleaning /plugins dir
  Processing plugin redhat/java11/latest
  Installing plugin extension 1/2
  Downloading plugin from https://download.jboss.org/jbosstools/vscode/3rdparty/vscode-java-debug/vscode-java-debug-0.24.0.vsix
  Installing plugin extension 2/2
  Downloading plugin from https://download.jboss.org/jbosstools/static/jdt.ls/stable/java-0.55.1-1984.vsix
  All plugin artifacts have been successfully downloaded
  Saving log of installed plugins
  pulling image "quay.io/eclipse/che-jwtproxy:dbd0578"
  Successfully pulled image "quay.io/eclipse/che-jwtproxy:dbd0578"
  Error: Error response from daemon: create 4d0496474e23b6248f552fe80f5b5018cfc745c373aaaf36fc133a9b0276a616: error while creating volume path '/var/lib/docker/volumes/4d0496474e23b6248f552fe80f5b5018cfc745c373aaaf36fc133a9b0276a616/_data': mkdir /var/lib/docker/volumes/4d0496474e23b6248f552fe80f5b5018cfc745c373aaaf36fc133a9b0276a616: permission denied

It looks like it is related to the jwt-proxy changes [1] and inconsistency with VOLUME in the dockerfile - https://github.com/eclipse/che-jwtproxy/blob/master/Dockerfile#L21 (in Che Server code the path was changed from /config/mykey.pub to /che-jwtproxy-config/mykey.pub)

The issue is currently a blocker for Hosted Che update to 7.9.0

[1] 193e64b#diff-691629a42d451ab233021b3ea7b5458fR44

Screenshots

Additional context

It is not clear why this issue is reproducible on Hosted Che dev-cluster environment, but was not caught in the upstream 🤷‍♂️

[ibuziuk]

Setting a blocker label since this issue is a showstopper for 7.9.0 update of Hosted Che - #16080

[ibuziuk]

@metlos works like a charm with CHE_SERVER_SECURE__EXPOSER_JWTPROXY_IMAGE: quay.io/lkrejci/che-jwtproxy:issue-15651

[ibuziuk]

@rhopp @dmytro-ndp there is an assumption that issue might be reproducible with common PVC strategy. Are we covering this case?

CHE_INFRA_KUBERNETES_PVC_STRATEGY: common

[dmytro-ndp]

We are not re-defining default CHE_INFRA_KUBERNETES_PVC_STRATEGY = common in time of Eclipse Che pre-release testing:
https://codeready-workspaces-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/basic-MultiUser-Che-check-e2e-tests-against-k8s/972/artifact/logs-and-configs/che-config/configmap.yaml/*view*/:

Che config

apiVersion: v1
data:
  CHE_API: http://che-che.10.0.101.47.nip.io/api
  CHE_DEBUG_SERVER: "false"
  CHE_HOST: che-che.10.0.101.47.nip.io
  CHE_INFRA_KUBERNETES_INGRESS_ANNOTATIONS__JSON: '{"kubernetes.io/ingress.class":
    nginx, "nginx.ingress.kubernetes.io/rewrite-target": "/$1","nginx.ingress.kubernetes.io/ssl-redirect":
    false,"nginx.ingress.kubernetes.io/proxy-connect-timeout": "3600","nginx.ingress.kubernetes.io/proxy-read-timeout":
    "3600"}'
  CHE_INFRA_KUBERNETES_INGRESS_DOMAIN: 10.0.101.47.nip.io
  CHE_INFRA_KUBERNETES_INGRESS_PATH__TRANSFORM: '%s(.*)'
  CHE_INFRA_KUBERNETES_NAMESPACE_ALLOW__USER__DEFINED: "false"
  CHE_INFRA_KUBERNETES_NAMESPACE_DEFAULT: che
  CHE_INFRA_KUBERNETES_POD_SECURITY__CONTEXT_FS__GROUP: "1724"
  CHE_INFRA_KUBERNETES_POD_SECURITY__CONTEXT_RUN__AS__USER: "1724"
  CHE_INFRA_KUBERNETES_PVC_JOBS_IMAGE: registry.access.redhat.com/ubi8-minimal:8.0-213
  CHE_INFRA_KUBERNETES_PVC_PRECREATE__SUBPATHS: "true"
  CHE_INFRA_KUBERNETES_PVC_QUANTITY: 1Gi
  CHE_INFRA_KUBERNETES_PVC_STORAGE__CLASS__NAME: ""
  CHE_INFRA_KUBERNETES_PVC_STRATEGY: common
  CHE_INFRA_KUBERNETES_SERVER__STRATEGY: multi-host
  CHE_INFRA_KUBERNETES_SERVICE__ACCOUNT__NAME: che-workspace
  CHE_INFRA_KUBERNETES_TLS__SECRET: ""
  CHE_INFRA_KUBERNETES_TRUST__CERTS: "false"
  CHE_INFRA_OPENSHIFT_OAUTH__IDENTITY__PROVIDER: "NULL"
  CHE_INFRA_OPENSHIFT_TLS__ENABLED: "false"
  CHE_INFRASTRUCTURE_ACTIVE: kubernetes
  CHE_JDBC_PASSWORD: PbWw8w3g7aFm
  CHE_JDBC_URL: jdbc:postgresql://postgres:5432/dbche
  CHE_JDBC_USERNAME: pgche
  CHE_KEYCLOAK_AUTH__SERVER__URL: http://keycloak-che.10.0.101.47.nip.io/auth
  CHE_KEYCLOAK_CLIENT__ID: che-public
  CHE_KEYCLOAK_REALM: che
  CHE_LOG_LEVEL: INFO
  CHE_METRICS_ENABLED: "false"
  CHE_MULTIUSER: "true"
  CHE_PORT: "8080"
  CHE_SERVER_SECURE__EXPOSER_JWTPROXY_IMAGE: quay.io/eclipse/che-jwtproxy:dbd0578
  CHE_WEBSOCKET_ENDPOINT: ws://che-che.10.0.101.47.nip.io/api/websocket
  CHE_WEBSOCKET_ENDPOINT__MINOR: ws://che-che.10.0.101.47.nip.io/api/websocket-minor
  CHE_WORKSPACE_DEVFILE__REGISTRY__URL: http://devfile-registry-che.10.0.101.47.nip.io
  CHE_WORKSPACE_HTTP__PROXY: ""
  CHE_WORKSPACE_HTTP__PROXY__JAVA__OPTIONS: ""
  CHE_WORKSPACE_HTTPS__PROXY: ""
  CHE_WORKSPACE_JAVA__OPTIONS: '-XX:MaxRAM=150m -XX:MaxRAMFraction=2 -XX:+UseParallelGC
    -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90
    -Dsun.zip.disableMemoryMapping=true -Xms20m -Djava.security.egd=file:/dev/./urandom '
  CHE_WORKSPACE_MAVEN__OPTIONS: '-XX:MaxRAM=150m -XX:MaxRAMFraction=2 -XX:+UseParallelGC
    -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90
    -Dsun.zip.disableMemoryMapping=true -Xms20m -Djava.security.egd=file:/dev/./urandom '
  CHE_WORKSPACE_NO__PROXY: ""
  CHE_WORKSPACE_PLUGIN__BROKER_ARTIFACTS_IMAGE: quay.io/eclipse/che-plugin-artifacts-broker:v3.1.0
  CHE_WORKSPACE_PLUGIN__BROKER_METADATA_IMAGE: quay.io/eclipse/che-plugin-metadata-broker:v3.1.0
  CHE_WORKSPACE_PLUGIN__REGISTRY__URL: http://plugin-registry-che.10.0.101.47.nip.io/v3
  JAVA_OPTS: '-XX:MaxRAMFraction=2 -XX:+UseParallelGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20
    -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:+UnlockExperimentalVMOptions
    -XX:+UseCGroupMemoryLimitForHeap -Dsun.zip.disableMemoryMapping=true -Xms20m '
  KUBERNETES_LABELS: app=che,component=che
kind: ConfigMap
metadata:
  creationTimestamp: "2020-02-21T15:40:29Z"
  labels:
    app: che
    component: che
  name: che
  namespace: che
  ownerReferences:
  - apiVersion: org.eclipse.che/v1
    blockOwnerDeletion: true
    controller: true
    kind: CheCluster
    name: eclipse-che
    uid: 3320e54a-54c0-11ea-aed4-fa163e86f637
  resourceVersion: "990"
  selfLink: /api/v1/namespaces/che/configmaps/che
  uid: 7cfafea8-54c0-11ea-aed4-fa163e86f637
 

[metlos]

I believe this is fixed by #16128.