Error configmaps (...)-gitconfig already exist when starting a workspace

[l0rd]

Describe the bug

Che version

• 7.9 (CRW 2.1.1)

Steps to reproduce

(still figuring that out...)

Runtime

• Openshift (include output of oc version)

Screenshots

Installation method

• OperatorHub

[skabashnyuk]

Duplicate #16758 ?

[l0rd]

Duplicate #16758 ?

Right, seems related. I have asked for the logs but I didn't get them yet. Some other information: "I have seen the error with 1 or 3 workspaces running. It happens after I leave the workspace hanging for a while."

[sparkoo]

I was able to reproduce. I don't have reliable reproduce steps. I was just randomly starting stopping 2 workspaces.

This is interesting:

2020-06-30 09:34:04,589[aceSharedPool-2]  [WARN ] [.i.k.KubernetesInternalRuntime 271]  - Failed to clean up namespace after workspace 'workspace368ycu0nhmwve9en' start failing. Cause: Error(s) occurs while cleaning up the namespace. Your session has expired. 
Please <a href='javascript:location.reload();' target='_top'>login</a> to Che again to get access to your OpenShift account Your session has expired. 
Please <a href='javascript:location.reload();' target='_top'>login</a> to Che again to get access to your OpenShift account Your session has expired. 
Please <a href='javascript:location.reload();' target='_top'>login</a> to Che again to get access to your OpenShift account Your session has expired. 
Please <a href='javascript:location.reload();' target='_top'>login</a> to Che again to get access to your OpenShift account Your session has expired. 
Please <a href='javascript:location.reload();' target='_top'>login</a> to Che again to get access to your OpenShift account
2020-06-30 09:34:04,595[aceSharedPool-2]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 934]   - Workspace 'sparkoo:wksp-y0rs' with id 'workspace368ycu0nhmwve9en' start failed
2020-06-30 09:51:32,104[nio-8080-exec-3]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 479]   - Starting workspace 'sparkoo/wksp-y0rs' with id 'workspace368ycu0nhmwve9en' by user 'sparkoo'
2020-06-30 09:54:02,291[ted-scheduler-2]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 540]   - Workspace 'sparkoo/wksp-hr6k' with id 'workspace6w7oan61xnuwwx3h' is stopping by user 'activity-checker'
2020-06-30 09:54:02,416[aceSharedPool-4]  [ERROR] [o.e.c.a.w.s.WorkspaceRuntimes 1003]  - Error occurred during stopping of runtime 'workspace6w7oan61xnuwwx3h:default:085be003-bafc-4a04-8e99-7ded4cb4c1b3' by user 'activity-checker'. Error: Error(s) occurs while cleaning up the namespace. Failure executing: GET at: https://172.30.0.1/apis/route.openshift.io/v1/namespaces/sparkoo-codeready/routes?labelSelector=che.workspace_id%3Dworkspace6w7oan61xnuwwx3h. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. routes.route.openshift.io is forbidden: User "system:serviceaccount:crw:che" cannot list resource "routes" in API group "route.openshift.io" in the namespace "sparkoo-codeready". The error may be caused by an expired token or changed password. Update Che server deployment with a new token or password. Failure executing: GET at: https://172.30.0.1/api/v1/namespaces/sparkoo-codeready/services?labelSelector=che.workspace_id%3Dworkspace6w7oan61xnuwwx3h. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. services is forbidden: User "system:serviceaccount:crw:che" cannot list resource "services" in API group "" in the namespace "sparkoo-codeready". The error may be caused by an expired token or changed password. Update Che server deployment with a new token or password. Failure executing: GET at: https://172.30.0.1/apis/apps/v1/namespaces/sparkoo-codeready/deployments?labelSelector=che.workspace_id%3Dworkspace6w7oan61xnuwwx3h. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. deployments.apps is forbidden: User "system:serviceaccount:crw:che" cannot list resource "deployments" in API group "apps" in the namespace "sparkoo-codeready". The error may be caused by an expired token or changed password. Update Che server deployment with a new token or password. Failure executing: GET at: https://172.30.0.1/api/v1/namespaces/sparkoo-codeready/secrets?labelSelector=che.workspace_id%3Dworkspace6w7oan61xnuwwx3h. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. secrets is forbidden: User "system:serviceaccount:crw:che" cannot list resource "secrets" in API group "" in the namespace "sparkoo-codeready". The error may be caused by an expired token or changed password. Update Che server deployment with a new token or password. Failure executing: GET at: https://172.30.0.1/api/v1/namespaces/sparkoo-codeready/configmaps?labelSelector=che.workspace_id%3Dworkspace6w7oan61xnuwwx3h. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. configmaps is forbidden: User "system:serviceaccount:crw:che" cannot list resource "configmaps" in API group "" in the namespace "sparkoo-codeready". The error may be caused by an expired token or changed password. Update Che server deployment with a new token or password.
org.eclipse.che.api.workspace.server.spi.InfrastructureException: Error(s) occurs while cleaning up the namespace. Failure executing: GET at: https://172.30.0.1/apis/route.openshift.io/v1/namespaces/sparkoo-codeready/routes?labelSelector=che.workspace_id%3Dworkspace6w7oan61xnuwwx3h. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. routes.route.openshift.io is forbidden: User "system:serviceaccount:crw:che" cannot list resource "routes" in API group "route.openshift.io" in the namespace "sparkoo-codeready". The error may be caused by an expired token or changed password. Update Che server deployment with a new token or password. Failure executing: GET at: https://172.30.0.1/api/v1/namespaces/sparkoo-codeready/services?labelSelector=che.workspace_id%3Dworkspace6w7oan61xnuwwx3h. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. services is forbidden: User "system:serviceaccount:crw:che" cannot list resource "services" in API group "" in the namespace "sparkoo-codeready". The error may be caused by an expired token or changed password. Update Che server deployment with a new token or password. Failure executing: GET at: https://172.30.0.1/apis/apps/v1/namespaces/sparkoo-codeready/deployments?labelSelector=che.workspace_id%3Dworkspace6w7oan61xnuwwx3h. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. deployments.apps is forbidden: User "system:serviceaccount:crw:che" cannot list resource "deployments" in API group "apps" in the namespace "sparkoo-codeready". The error may be caused by an expired token or changed password. Update Che server deployment with a new token or password. Failure executing: GET at: https://172.30.0.1/api/v1/namespaces/sparkoo-codeready/secrets?labelSelector=che.workspace_id%3Dworkspace6w7oan61xnuwwx3h. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. secrets is forbidden: User "system:serviceaccount:crw:che" cannot list resource "secrets" in API group "" in the namespace "sparkoo-codeready". The error may be caused by an expired token or changed password. Update Che server deployment with a new token or password. Failure executing: GET at: https://172.30.0.1/api/v1/namespaces/sparkoo-codeready/configmaps?labelSelector=che.workspace_id%3Dworkspace6w7oan61xnuwwx3h. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. configmaps is forbidden: User "system:serviceaccount:crw:che" cannot list resource "configmaps" in API group "" in the namespace "sparkoo-codeready". The error may be caused by an expired token or changed password. Update Che server deployment with a new token or password.
	at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesNamespace.doRemove(KubernetesNamespace.java:270)
	at org.eclipse.che.workspace.infrastructure.openshift.project.OpenShiftProject.cleanUp(OpenShiftProject.java:160)
	at org.eclipse.che.workspace.infrastructure.kubernetes.KubernetesInternalRuntime.internalStop(KubernetesInternalRuntime.java:579)
	at org.eclipse.che.api.workspace.server.spi.InternalRuntime.stop(InternalRuntime.java:177)
	at org.eclipse.che.api.workspace.server.WorkspaceRuntimes$StopRuntimeTask.run(WorkspaceRuntimes.java:973)
	at org.eclipse.che.commons.lang.concurrent.CopyThreadLocalRunnable.run(CopyThreadLocalRunnable.java:38)
	at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1640)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

[sparkoo]

I think the issue should be already fixed in upstream Che by this #16498. To confirm that, I'm interested if second workspace start is ok.

[sparkoo]

@l0rd we can backport or just suggest to wait for new CRW version.

[l0rd]

@sparkoo is #16498 included in 7.14.x? If that's the case we are all set because it will be included in CRW 2.2. Otherwise we should consider porting it in 7.14.x before the 2.2 release.

[sparkoo]

@l0rd yes. it's in since 7.11

[skabashnyuk]

@l0rd can we close this issue or do you expect any other activity from our side?

[skabashnyuk]

Fixed in 7.12 in #16418

[nickboldt]

I see the "ask a question, wait a week, assume it's done" approach being used here. :)