Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error after upgrade from 7.3.3 to 7.9.2 #16612

Closed
3 of 23 tasks
skabashnyuk opened this issue Apr 15, 2020 · 5 comments
Closed
3 of 23 tasks

Error after upgrade from 7.3.3 to 7.9.2 #16612

skabashnyuk opened this issue Apr 15, 2020 · 5 comments
Assignees
@metlos @skabashnyuk
Labels
area/che-server kind/bug Outline of a bug - must adhere to the bug report template. severity/P1 Has a major impact to usage or development of the system.
Projects
Che Platform team sprint #182
Milestone
7.12

Comments

@skabashnyuk
Copy link
Contributor

skabashnyuk commented Apr 15, 2020 •
edited
Loading

Describe the bug

image

Error: Failed to start the workspace: "Failure executing: PUT at: https://172.30.0.1/apis/project.openshift.io/v1/projects/workspacelzjvluj3yhaga0ix. Message: Project.project.openshift.io "workspacelzjvluj3yhaga0ix" is invalid: metadata.labels[che-managed]: Invalid value: "true": field is immutable, , try updating the namespace. Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[StatusCause(field=metadata.labels[che-managed], message=Invalid value: "true": field is immutable, , try updating the namespace, reason=FieldValueInvalid, additionalProperties={})], group=project.openshift.io, kind=Project, name=workspacelzjvluj3yhaga0ix, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=Project.project.openshift.io "workspacelzjvluj3yhaga0ix" is invalid: metadata.labels[che-managed]: Invalid value: "true": field is immutable, , try updating the namespace, metadata=ListMeta(_continue=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure, additionalProperties={})."

Che version

  • latest
  • nightly
  • other:7.9.2

Steps to reproduce

  1. 7.3.3 workspace created in the individual namespace.
  2. upgrade to 7.9.2
  3. Start existed workspace.

Expected behavior

Workspace should start without error.

Runtime

  • kubernetes (include output of kubectl version)
  • Openshift (include output of oc version)
  • minikube (include output of minikube version and kubectl version)
  • minishift (include output of minishift version and oc version)
  • docker-desktop + K8S (include output of docker version and kubectl version)
  • other: (please specify)

Screenshots

n/a

Installation method

  • chectl
  • che-operator
  • minishift-addon
  • I don't know

Environment

  • my computer
    • Windows
    • Linux
    • macOS
  • Cloud
    • Amazon
    • Azure
    • GCE
    • other (please specify)
  • other: please specify

Eclipse Che Logs

Additional context

che config map of the 2.0.0 Che server install with default settings + TLS (=> Openshift OAuth + TLS + selfsigned certs): 

CHE_INFRA_KUBERNETES_PVC_PRECREATE__SUBPATHS: 'true'
  CHE_PORT: '8080'
  CHE_INFRA_KUBERNETES_PVC_STRATEGY: per-workspace
  CHE_JDBC_PASSWORD: hDw1kyi6nke2
  CHE_KEYCLOAK_REALM: codeready
  CHE_INFRA_KUBERNETES_TRUST__CERTS: 'true'
  CHE_INFRASTRUCTURE_ACTIVE: openshift
  CHE_WORKSPACE_NO__PROXY: ''
  CHE_WORKSPACE_PLUGIN__BROKER_UNIFIED_IMAGE: 'registry.redhat.io/codeready-workspaces/pluginbroker-rhel8:2.0'
  CHE_INFRA_OPENSHIFT_PROJECT: ''
  CHE_MULTIUSER: 'true'
  CHE_HOST: codeready-test-david-prod-update.apps.test-ocp43.codereadyqe.com
  CHE_SERVER_SECURE__EXPOSER_JWTPROXY_IMAGE: 'registry.redhat.io/codeready-workspaces/jwtproxy-rhel8:2.0'
  CHE_WEBSOCKET_ENDPOINT__MINOR: >-
    wss://codeready-test-david-prod-update.apps.test-ocp43.codereadyqe.com/api/websocket-minor
  CHE_LOG_LEVEL: INFO
  CHE_JDBC_URL: 'jdbc:postgresql://postgres:5432/dbche'
  CHE_WORKSPACE_DEVFILE__REGISTRY__URL: >-
    https://devfile-registry-test-david-prod-update.apps.test-ocp43.codereadyqe.com
  CHE_WEBSOCKET_ENDPOINT: >-
    wss://codeready-test-david-prod-update.apps.test-ocp43.codereadyqe.com/api/websocket
  CHE_WORKSPACE_HTTPS__PROXY: ''
  CHE_DEBUG_SERVER: 'false'
  CHE_INFRA_KUBERNETES_SERVICE__ACCOUNT__NAME: che-workspace
  CHE_INFRA_OPENSHIFT_OAUTH__IDENTITY__PROVIDER: openshift-v4
  CHE_WORKSPACE_JAVA__OPTIONS: >-
    -XX:MaxRAM=150m -XX:MaxRAMFraction=2 -XX:+UseParallelGC
    -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4
    -XX:AdaptiveSizePolicyWeight=90 -Dsun.zip.disableMemoryMapping=true -Xms20m
    -Djava.security.egd=file:/dev/./urandom 
  CHE_KEYCLOAK_CLIENT__ID: codeready-public
  CHE_INFRA_KUBERNETES_PVC_STORAGE__CLASS__NAME: ''
  CHE_KEYCLOAK_AUTH__SERVER__URL: 'https://keycloak-test-david-prod-update.apps.test-ocp43.codereadyqe.com/auth'
  CHE_WORKSPACE_PLUGIN__REGISTRY__URL: >-
    https://plugin-registry-test-david-prod-update.apps.test-ocp43.codereadyqe.com/v3
  CHE_WORKSPACE_PLUGIN__BROKER_INIT_IMAGE: 'registry.redhat.io/codeready-workspaces/pluginbrokerinit-rhel8:2.0'
  CHE_INFRA_KUBERNETES_PVC_QUANTITY: 1Gi
  CHE_JDBC_USERNAME: pgche
  CHE_INFRA_OPENSHIFT_TLS__ENABLED: 'true'
  KUBERNETES_LABEL: 'app=codeready,component=codeready'
  CHE_API: 'https://codeready-test-david-prod-update.apps.test-ocp43.codereadyqe.com/api'
  CHE_WORKSPACE_MAVEN__OPTIONS: >-
    -XX:MaxRAM=150m -XX:MaxRAMFraction=2 -XX:+UseParallelGC
    -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4
    -XX:AdaptiveSizePolicyWeight=90 -Dsun.zip.disableMemoryMapping=true -Xms20m
    -Djava.security.egd=file:/dev/./urandom 
  JAVA_OPTS: >-
    -XX:MaxRAMFraction=2 -XX:+UseParallelGC -XX:MinHeapFreeRatio=10
    -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90
    -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap
    -Dsun.zip.disableMemoryMapping=true -Xms20m 
  CHE_WORKSPACE_HTTP__PROXY: ''
  CHE_INFRA_KUBERNETES_PVC_JOBS_IMAGE: 'registry.redhat.io/ubi8-minimal:8.0-213'
  CHE_WORKSPACE_HTTP__PROXY__JAVA__OPTIONS: ''

l Che server POD env variables

    env:
        - name: CM_REVISION
          value: '4727172'
        - name: KUBERNETES_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: CHE_SELF__SIGNED__CERT
          valueFrom:
            secretKeyRef:
              name: self-signed-certificate
              key: ca.crt
              optional: true

Now Config map of the 2.1.0 Che server:: 

CHE_INFRA_KUBERNETES_PVC_PRECREATE__SUBPATHS: 'true'
  CHE_PORT: '8080'
  CHE_INFRA_KUBERNETES_PVC_STRATEGY: per-workspace
  CHE_JDBC_PASSWORD: hDw1kyi6nke2
  CHE_KEYCLOAK_REALM: codeready
  CHE_INFRA_KUBERNETES_TRUST__CERTS: 'true'
  CHE_INFRASTRUCTURE_ACTIVE: openshift
  CHE_WORKSPACE_NO__PROXY: ''
  CHE_INFRA_KUBERNETES_NAMESPACE_ALLOW__USER__DEFINED: 'false'
  CHE_MULTIUSER: 'true'
  CHE_METRICS_ENABLED: 'false'
  CHE_HOST: codeready-test-david-prod-update.apps.test-ocp43.codereadyqe.com
  CHE_SERVER_SECURE__EXPOSER_JWTPROXY_IMAGE: >-
    registry.redhat.io/codeready-workspaces/jwtproxy-rhel8@sha256:63182dae6377ac01fbebdcb9c6a4435681ea88521cfc6e4222fe3920a3641127
  CHE_WEBSOCKET_ENDPOINT__MINOR: >-
    wss://codeready-test-david-prod-update.apps.test-ocp43.codereadyqe.com/api/websocket-minor
  CHE_LOG_LEVEL: INFO
  CHE_JDBC_URL: 'jdbc:postgresql://postgres:5432/dbche'
  CHE_WORKSPACE_DEVFILE__REGISTRY__URL: >-
    https://devfile-registry-test-david-prod-update.apps.test-ocp43.codereadyqe.com
  CHE_WEBSOCKET_ENDPOINT: >-
    wss://codeready-test-david-prod-update.apps.test-ocp43.codereadyqe.com/api/websocket
  CHE_WORKSPACE_HTTPS__PROXY: ''
  CHE_DEBUG_SERVER: 'false'
  CHE_INFRA_KUBERNETES_SERVICE__ACCOUNT__NAME: che-workspace
  CHE_INFRA_OPENSHIFT_OAUTH__IDENTITY__PROVIDER: openshift-v4
  CHE_INFRA_KUBERNETES_NAMESPACE_DEFAULT: <username>-codeready
  CHE_WORKSPACE_JAVA__OPTIONS: >-
    -XX:MaxRAM=150m -XX:MaxRAMFraction=2 -XX:+UseParallelGC
    -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4
    -XX:AdaptiveSizePolicyWeight=90 -Dsun.zip.disableMemoryMapping=true -Xms20m
    -Djava.security.egd=file:/dev/./urandom 
  CHE_WORKSPACE_PLUGIN__BROKER_ARTIFACTS_IMAGE: >-
    registry.redhat.io/codeready-workspaces/pluginbroker-artifacts-rhel8@sha256:5815bab69fc343cbf6dac0fd67dd70a25757fac08689a15e4a762655fa2e8a2c
  CHE_KEYCLOAK_CLIENT__ID: codeready-public
  CHE_WORKSPACE_PLUGIN__BROKER_METADATA_IMAGE: >-
    registry.redhat.io/codeready-workspaces/pluginbroker-metadata-rhel8@sha256:6c9abe63a70a6146dc49845f2f7732e3e6e0bcae6a19c3a6557367d6965bc1f8
  CHE_INFRA_KUBERNETES_PVC_STORAGE__CLASS__NAME: ''
  CHE_KEYCLOAK_AUTH__SERVER__URL: 'https://keycloak-test-david-prod-update.apps.test-ocp43.codereadyqe.com/auth'
  CHE_WORKSPACE_PLUGIN__REGISTRY__URL: >-
    https://plugin-registry-test-david-prod-update.apps.test-ocp43.codereadyqe.com/v3
  CHE_INFRA_KUBERNETES_PVC_QUANTITY: 1Gi
  KUBERNETES_LABELS: 'app=codeready,component=codeready'
  CHE_JDBC_USERNAME: pgche
  CHE_INFRA_OPENSHIFT_TLS__ENABLED: 'true'
  CHE_API: 'https://codeready-test-david-prod-update.apps.test-ocp43.codereadyqe.com/api'
  CHE_WORKSPACE_MAVEN__OPTIONS: >-
    -XX:MaxRAM=150m -XX:MaxRAMFraction=2 -XX:+UseParallelGC
    -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4
    -XX:AdaptiveSizePolicyWeight=90 -Dsun.zip.disableMemoryMapping=true -Xms20m
    -Djava.security.egd=file:/dev/./urandom 
  JAVA_OPTS: >-
    -XX:MaxRAMFraction=2 -XX:+UseParallelGC -XX:MinHeapFreeRatio=10
    -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90
    -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap
    -Dsun.zip.disableMemoryMapping=true -Xms20m 
  CHE_WORKSPACE_HTTP__PROXY: ''
  CHE_INFRA_KUBERNETES_PVC_JOBS_IMAGE: >-
    registry.access.redhat.com/ubi8-minimal@sha256:9285da611437622492f9ef4229877efe302589f1401bbd4052e9bb261b3d4387
  CHE_WORKSPACE_HTTP__PROXY__JAVA__OPTIONS: ''

  - name: CM_REVISION
          value: '4732509'
        - name: KUBERNETES_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: CHE_SELF__SIGNED__CERT
          valueFrom:
            secretKeyRef:
              name: self-signed-certificate
              key: ca.crt
              optional: true
        - name: CHE_GIT_SELF__SIGNED__CERT
        - name: CHE_GIT_SELF__SIGNED__CERT__HOST

Che server logs 


15-Apr-2020 12:51:47.844 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 13244 ms
2020-04-15 12:53:53,027[nio-8080-exec-4]  [ERROR] [o.e.c.a.w.s.WorkspaceRuntimes 494]   - Failure executing: PUT at: https://172.30.0.1/apis/project.openshift.io/v1/projects/workspace0fyulpv8u0w7ne3s. Message: Project.project.openshift.io "workspace0fyulpv8u0w7ne3s" is invalid: metadata.labels[che-managed]: Invalid value: "true": field is immutable, , try updating the namespace. Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[StatusCause(field=metadata.labels[che-managed], message=Invalid value: "true": field is immutable, , try updating the namespace, reason=FieldValueInvalid, additionalProperties={})], group=project.openshift.io, kind=Project, name=workspace0fyulpv8u0w7ne3s, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=Project.project.openshift.io "workspace0fyulpv8u0w7ne3s" is invalid: metadata.labels[che-managed]: Invalid value: "true": field is immutable, , try updating the namespace, metadata=ListMeta(_continue=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure, additionalProperties={}).
org.eclipse.che.workspace.infrastructure.kubernetes.KubernetesInfrastructureException: Failure executing: PUT at: https://172.30.0.1/apis/project.openshift.io/v1/projects/workspace0fyulpv8u0w7ne3s. Message: Project.project.openshift.io "workspace0fyulpv8u0w7ne3s" is invalid: metadata.labels[che-managed]: Invalid value: "true": field is immutable, , try updating the namespace. Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[StatusCause(field=metadata.labels[che-managed], message=Invalid value: "true": field is immutable, , try updating the namespace, reason=FieldValueInvalid, additionalProperties={})], group=project.openshift.io, kind=Project, name=workspace0fyulpv8u0w7ne3s, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=Project.project.openshift.io "workspace0fyulpv8u0w7ne3s" is invalid: metadata.labels[che-managed]: Invalid value: "true": field is immutable, , try updating the namespace, metadata=ListMeta(_continue=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure, additionalProperties={}).
	at org.eclipse.che.workspace.infrastructure.openshift.project.OpenShiftProject.update(OpenShiftProject.java:193)
	at org.eclipse.che.workspace.infrastructure.openshift.project.OpenShiftProject.prepare(OpenShiftProject.java:120)
	at org.eclipse.che.workspace.infrastructure.openshift.project.OpenShiftProjectFactory.getOrCreate(OpenShiftProjectFactory.java:86)
	at org.eclipse.che.workspace.infrastructure.openshift.OpenShiftRuntimeContext.getRuntime(OpenShiftRuntimeContext.java:69)
	at org.eclipse.che.workspace.infrastructure.openshift.OpenShiftRuntimeContext.getRuntime(OpenShiftRuntimeContext.java:31)
	at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.startAsync(WorkspaceRuntimes.java:466)
	at org.eclipse.che.api.workspace.server.WorkspaceManager.startAsync(WorkspaceManager.java:488)
	at org.eclipse.che.api.workspace.server.WorkspaceManager.startWorkspace(WorkspaceManager.java:373)
	at org.eclipse.che.multiuser.resource.api.workspace.LimitsCheckingWorkspaceManager.startWorkspace(LimitsCheckingWorkspaceManager.java:132)
	at org.eclipse.che.api.workspace.server.WorkspaceService.startById(WorkspaceService.java:458)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.everrest.core.impl.method.DefaultMethodInvoker.invokeMethod(DefaultMethodInvoker.java:140)
	at org.everrest.core.impl.method.DefaultMethodInvoker.invokeMethod(DefaultMethodInvoker.java:60)
	at org.everrest.core.impl.RequestDispatcher.doInvokeResource(RequestDispatcher.java:306)
	at org.everrest.core.impl.RequestDispatcher.invokeSubResourceMethod(RequestDispatcher.java:297)
	at org.everrest.core.impl.RequestDispatcher.dispatch(RequestDispatcher.java:233)
	at org.everrest.core.impl.RequestDispatcher.dispatch(RequestDispatcher.java:128)
	at org.everrest.core.impl.RequestHandlerImpl.handleRequest(RequestHandlerImpl.java:62)
	at org.everrest.core.impl.EverrestProcessor.process(EverrestProcessor.java:120)
	at org.everrest.core.servlet.EverrestServlet.service(EverrestServlet.java:61)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
	at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:290)
	at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:280)
	at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:184)
	at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:89)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
	at org.eclipse.che.commons.logback.filter.IdentityIdLoggerFilter.doFilter(IdentityIdLoggerFilter.java:49)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at org.eclipse.che.multiuser.api.authentication.commons.filter.MultiUserEnvironmentInitializationFilter.doFilter(MultiUserEnvironmentInitializationFilter.java:142)
	at org.eclipse.che.multiuser.keycloak.server.KeycloakEnvironmentInitializationFilter.doFilter(KeycloakEnvironmentInitializationFilter.java:88)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at org.eclipse.che.multiuser.machine.authentication.server.MachineLoginFilter.doFilter(MachineLoginFilter.java:76)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at org.eclipse.che.commons.logback.filter.RequestIdLoggerFilter.doFilter(RequestIdLoggerFilter.java:50)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:121)
	at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:528)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
	at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:747)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:798)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:810)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1500)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:748)
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PUT at: https://172.30.0.1/apis/project.openshift.io/v1/projects/workspace0fyulpv8u0w7ne3s. Message: Project.project.openshift.io "workspace0fyulpv8u0w7ne3s" is invalid: metadata.labels[che-managed]: Invalid value: "true": field is immutable, , try updating the namespace. Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[StatusCause(field=metadata.labels[che-managed], message=Invalid value: "true": field is immutable, , try updating the namespace, reason=FieldValueInvalid, additionalProperties={})], group=project.openshift.io, kind=Project, name=workspace0fyulpv8u0w7ne3s, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=Project.project.openshift.io "workspace0fyulpv8u0w7ne3s" is invalid: metadata.labels[che-managed]: Invalid value: "true": field is immutable, , try updating the namespace, metadata=ListMeta(_continue=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure, additionalProperties={}).
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:476)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:415)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:381)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:344)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleReplace(OperationSupport.java:261)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleReplace(OperationSupport.java:243)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleReplace(BaseOperation.java:785)
	at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation$2.apply(HasMetadataOperation.java:109)
	at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation$2.apply(HasMetadataOperation.java:104)
	at io.fabric8.openshift.api.model.DoneableProject.done(DoneableProject.java:27)
	at io.fabric8.openshift.api.model.DoneableProject.done(DoneableProject.java:6)
	at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:116)
	at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:37)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:420)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:415)
	at org.eclipse.che.workspace.infrastructure.openshift.project.OpenShiftProject.update(OpenShiftProject.java:186)
	... 58 common frames omitted

@skabashnyuk skabashnyuk added kind/bug Outline of a bug - must adhere to the bug report template. area/che-server severity/P1 Has a major impact to usage or development of the system. team/platform labels Apr 15, 2020
@skabashnyuk skabashnyuk added this to the Backlog - Platform milestone Apr 15, 2020
@skabashnyuk
Copy link
Contributor Author

Reported by @davidfestal

@nickboldt nickboldt modified the milestones: Backlog - Platform, 7.12 Apr 15, 2020
@skabashnyuk
Copy link
Contributor Author

Reproduced on crc with

  server:
    customCheProperties:
      CHE_INFRA_KUBERNETES_NAMESPACE_DEFAULT: <workspaceid>-che

2020-04-15 14:34:01,672[nio-8080-exec-4]  [ERROR] [o.e.c.a.w.s.WorkspaceRuntimes 494]   - Failure executing: PUT at: https://172.30.0.1/apis/project.openshift.io/v1/projects/workspacehtmavrhfu2p03p6j-che. Message: Project.project.openshift.io "workspacehtmavrhfu2p03p6j-che" is invalid: metadata.labels[che-managed]: Invalid value: "true": field is immutable, , try updating the namespace. Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[StatusCause(field=metadata.labels[che-managed], message=Invalid value: "true": field is immutable, , try updating the namespace, reason=FieldValueInvalid, additionalProperties={})], group=project.openshift.io, kind=Project, name=workspacehtmavrhfu2p03p6j-che, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=Project.project.openshift.io "workspacehtmavrhfu2p03p6j-che" is invalid: metadata.labels[che-managed]: Invalid value: "true": field is immutable, , try updating the namespace, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure, additionalProperties={}).
org.eclipse.che.workspace.infrastructure.kubernetes.KubernetesInfrastructureException: Failure executing: PUT at: https://172.30.0.1/apis/project.openshift.io/v1/projects/workspacehtmavrhfu2p03p6j-che. Message: Project.project.openshift.io "workspacehtmavrhfu2p03p6j-che" is invalid: metadata.labels[che-managed]: Invalid value: "true": field is immutable, , try updating the namespace. Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[StatusCause(field=metadata.labels[che-managed], message=Invalid value: "true": field is immutable, , try updating the namespace, reason=FieldValueInvalid, additionalProperties={})], group=project.openshift.io, kind=Project, name=workspacehtmavrhfu2p03p6j-che, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=Project.project.openshift.io "workspacehtmavrhfu2p03p6j-che" is invalid: metadata.labels[che-managed]: Invalid value: "true": field is immutable, , try updating the namespace, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure, additionalProperties={}).
	at org.eclipse.che.workspace.infrastructure.openshift.project.OpenShiftProject.update(OpenShiftProject.java:196)
	at org.eclipse.che.workspace.infrastructure.openshift.project.OpenShiftProject.prepare(OpenShiftProject.java:123)
	at org.eclipse.che.workspace.infrastructure.openshift.project.OpenShiftProjectFactory.getOrCreate(OpenShiftProjectFactory.java:89)
	at org.eclipse.che.workspace.infrastructure.openshift.OpenShiftRuntimeContext.getRuntime(OpenShiftRuntimeContext.java:69)
	at org.eclipse.che.workspace.infrastructure.openshift.OpenShiftRuntimeContext.getRuntime(OpenShiftRuntimeContext.java:31)
	at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.startAsync(WorkspaceRuntimes.java:466)
	at org.eclipse.che.api.workspace.server.WorkspaceManager.startAsync(WorkspaceManager.java:488)
	at org.eclipse.che.api.workspace.server.WorkspaceManager.startWorkspace(WorkspaceManager.java:373)
	at org.eclipse.che.multiuser.resource.api.workspace.LimitsCheckingWorkspaceManager.startWorkspace(LimitsCheckingWorkspaceManager.java:132)
	at org.eclipse.che.api.workspace.server.WorkspaceService.startById(WorkspaceService.java:469)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.everrest.core.impl.method.DefaultMethodInvoker.invokeMethod(DefaultMethodInvoker.java:140)
	at org.everrest.core.impl.method.DefaultMethodInvoker.invokeMethod(DefaultMethodInvoker.java:60)
	at org.everrest.core.impl.RequestDispatcher.doInvokeResource(RequestDispatcher.java:306)
	at org.everrest.core.impl.RequestDispatcher.invokeSubResourceMethod(RequestDispatcher.java:297)
	at org.everrest.core.impl.RequestDispatcher.dispatch(RequestDispatcher.java:233)
	at org.everrest.core.impl.RequestDispatcher.dispatch(RequestDispatcher.java:128)
	at org.everrest.core.impl.RequestHandlerImpl.handleRequest(RequestHandlerImpl.java:62)
	at org.everrest.core.impl.EverrestProcessor.process(EverrestProcessor.java:120)
	at org.everrest.core.servlet.EverrestServlet.service(EverrestServlet.java:61)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
	at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:290)
	at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:280)
	at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:184)
	at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:89)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
	at org.eclipse.che.commons.logback.filter.IdentityIdLoggerFilter.doFilter(IdentityIdLoggerFilter.java:49)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at org.eclipse.che.multiuser.api.authentication.commons.filter.MultiUserEnvironmentInitializationFilter.doFilter(MultiUserEnvironmentInitializationFilter.java:142)
	at org.eclipse.che.multiuser.keycloak.server.KeycloakEnvironmentInitializationFilter.doFilter(KeycloakEnvironmentInitializationFilter.java:88)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at org.eclipse.che.multiuser.machine.authentication.server.MachineLoginFilter.doFilter(MachineLoginFilter.java:76)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at org.eclipse.che.commons.logback.filter.RequestIdLoggerFilter.doFilter(RequestIdLoggerFilter.java:50)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:121)
	at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:543)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
	at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:747)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:609)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:818)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1623)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:748)
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PUT at: https://172.30.0.1/apis/project.openshift.io/v1/projects/workspacehtmavrhfu2p03p6j-che. Message: Project.project.openshift.io "workspacehtmavrhfu2p03p6j-che" is invalid: metadata.labels[che-managed]: Invalid value: "true": field is immutable, , try updating the namespace. Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[StatusCause(field=metadata.labels[che-managed], message=Invalid value: "true": field is immutable, , try updating the namespace, reason=FieldValueInvalid, additionalProperties={})], group=project.openshift.io, kind=Project, name=workspacehtmavrhfu2p03p6j-che, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=Project.project.openshift.io "workspacehtmavrhfu2p03p6j-che" is invalid: metadata.labels[che-managed]: Invalid value: "true": field is immutable, , try updating the namespace, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure, additionalProperties={}).
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:568)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:507)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:471)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:430)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleReplace(OperationSupport.java:289)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleReplace(OperationSupport.java:269)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleReplace(BaseOperation.java:820)
	at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.lambda$replace$1(HasMetadataOperation.java:86)
	at io.fabric8.openshift.api.model.DoneableProject.done(DoneableProject.java:27)
	at io.fabric8.openshift.api.model.DoneableProject.done(DoneableProject.java:6)
	at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:92)
	at io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:36)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:396)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:391)
	at org.eclipse.che.workspace.infrastructure.openshift.project.OpenShiftProject.update(OpenShiftProject.java:189)
	... 58 common frames omitted

@nickboldt
Copy link
Contributor

For my reference since this is a wsmaster issue, the affected container is the che server one.

That means we need a 7.9.3 release since today CRW build relies on a full release of Che -- cannot use snapshots.

@skabashnyuk
Copy link
Contributor Author

ok. We are going to provide cherry-pick to 7.9.x as soon as fix would be available for master

@nickboldt nickboldt mentioned this issue Apr 15, 2020
Closed
14 tasks
@metlos
Copy link
Contributor

metlos commented Apr 16, 2020

PR #16617 handles this quite nicely.

@davidfestal davidfestal mentioned this issue Apr 16, 2020
Merged
metlos added a commit to metlos/che that referenced this issue Apr 16, 2020
@metlos
Fixes eclipse-che#16612 - Don't use the label to designate managed na…
65096eb 
…mespaces. Instead

rely solely on the heuristics.

Signed-off-by: Lukas Krejci <lkrejci@redhat.com>
@metlos metlos mentioned this issue Apr 16, 2020
Merged
@metlos metlos closed this as completed in 06a34e3 Apr 20, 2020
@metlos metlos added this to To do in Che Platform team sprint #182 via automation Apr 20, 2020
@metlos metlos moved this from To do to Done in Che Platform team sprint #182 Apr 20, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@metlos metlos

@skabashnyuk skabashnyuk

Labels
area/che-server kind/bug Outline of a bug - must adhere to the bug report template. severity/P1 Has a major impact to usage or development of the system.
Projects
No open projects
Che Platform team sprint #182
  
Done
Milestone
7.12
Development

No branches or pull requests
3 participants
@metlos @nickboldt @skabashnyuk