Error occurred while deleting snapshot

[skryzhny]

Got this error:

2016-12-12 15:29:41,901[kspaceManager-2]  [INFO ] [o.e.c.a.w.s.WorkspaceManager 705]    - Workspace '******:******' with id '1111111111111111111111' stopped by user '22222222222222222222'
2016-12-12 15:29:41,934[nio-8080-exec-8]  [INFO ] [arerTokenAuthenticationService 153]  - EVENT#user-sso-logged-in# USING#google# USER#33333333333333333333#
2016-12-12 15:29:42,534[io-8080-exec-23]  [ERROR] [c.p.d.m.DockerInstanceProvider 168]  - An error occurred while deleting snapshot with url: http://codenvy.io:5000/v2/machine_snapshot_44444444444444444444444444/manifests/sha256:555555555555555555555555555555
Error stream: {"errors":[{"code":"MANIFEST_UNKNOWN","message":"manifest unknown"}]}

2016-12-12 15:29:42,535[io-8080-exec-23]  [ERROR] [o.e.c.a.w.s.WorkspaceManager 568]    - Internal server error occurs. Can't remove snapshot
org.eclipse.che.api.machine.server.exception.SnapshotException: Internal server error occurs. Can't remove snapshot
        at org.eclipse.che.plugin.docker.machine.DockerInstanceProvider.removeInstanceSnapshot(DockerInstanceProvider.java:171) ~[che-plugin-docker-machine-5.0.0-M8.jar:5.0.0-M8]
        at org.eclipse.che.api.environment.server.CheEnvironmentEngine.removeSnapshot(CheEnvironmentEngine.java:528) ~[che-core-api-workspace-5.0.0-M8.jar:5.0.0-M8]
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.removeSnapshot(WorkspaceRuntimes.java:494) ~[che-core-api-workspace-5.0.0-M8.jar:5.0.0-M8]
        at org.eclipse.che.api.workspace.server.WorkspaceManager.removeSnapshots(WorkspaceManager.java:565) ~[che-core-api-workspace-5.0.0-M8.jar:5.0.0-M8]
        at org.eclipse.che.api.workspace.server.WorkspaceService.delete(WorkspaceService.java:285) [che-core-api-workspace-5.0.0-M8.jar:5.0.0-M8]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_45]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_45]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_45]
        at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_45]
        at org.everrest.core.impl.method.DefaultMethodInvoker.invokeMethod(DefaultMethodInvoker.java:140) [everrest-core-1.13.1.jar:na]
        at org.everrest.core.impl.method.DefaultMethodInvoker.invokeMethod(DefaultMethodInvoker.java:60) [everrest-core-1.13.1.jar:na]
        at org.everrest.core.impl.RequestDispatcher.doInvokeResource(RequestDispatcher.java:306) [everrest-core-1.13.1.jar:na]
        at org.everrest.core.impl.RequestDispatcher.invokeSubResourceMethod(RequestDispatcher.java:297) [everrest-core-1.13.1.jar:na]
        at org.everrest.core.impl.RequestDispatcher.dispatch(RequestDispatcher.java:233) [everrest-core-1.13.1.jar:na]
        at org.everrest.core.impl.RequestDispatcher.dispatch(RequestDispatcher.java:128) [everrest-core-1.13.1.jar:na]
        at org.everrest.core.impl.RequestHandlerImpl.handleRequest(RequestHandlerImpl.java:62) [everrest-core-1.13.1.jar:na]
        at org.everrest.core.impl.EverrestProcessor.process(EverrestProcessor.java:120) [everrest-core-1.13.1.jar:na]
        at org.everrest.core.servlet.EverrestServlet.service(EverrestServlet.java:61) [everrest-core-1.13.1.jar:na]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) [servlet-api.jar:na]
        at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:286) [guice-servlet-4.1.0.jar:na]
        at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:276) [guice-servlet-4.1.0.jar:na]
        at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:181) [guice-servlet-4.1.0.jar:na]
        at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91) [guice-servlet-4.1.0.jar:na]
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85) [guice-servlet-4.1.0.jar:na]
        at org.apache.catalina.filters.CorsFilter.handleNonCORS(CorsFilter.java:436) [catalina.jar:8.0.38]
        at org.apache.catalina.filters.CorsFilter.doFilter(CorsFilter.java:177) [catalina.jar:8.0.38]
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) [guice-servlet-4.1.0.jar:na]
        at com.codenvy.auth.sso.client.DefaultTokenHandler.handleValidToken(DefaultTokenHandler.java:65) [codenvy-hosted-sso-client-5.0.0-M8.jar:5.0.0-M8]
        at com.codenvy.api.permission.server.PermissionTokenHandler.handleValidToken(PermissionTokenHandler.java:64) [codenvy-hosted-authorization-5.0.0-M8.jar:5.0.0-M8]
        at com.codenvy.auth.sso.client.LoginFilter.doFilter(LoginFilter.java:120) [codenvy-hosted-sso-client-5.0.0-M8.jar:5.0.0-M8]
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) [guice-servlet-4.1.0.jar:na]
        at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:120) [guice-servlet-4.1.0.jar:na]
        at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:135) [guice-servlet-4.1.0.jar:na]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) [catalina.jar:8.0.38]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) [catalina.jar:8.0.38]
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212) [catalina.jar:8.0.38]
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106) [catalina.jar:8.0.38]
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141) [catalina.jar:8.0.38]
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79) [catalina.jar:8.0.38]
        at ch.qos.logback.access.tomcat.LogbackValve.invoke(LogbackValve.java:189) [logback-access-1.1.2.jar:na]
        at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:676) [catalina.jar:8.0.38]
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88) [catalina.jar:8.0.38]
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:528) [catalina.jar:8.0.38]
        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1100) [tomcat-coyote.jar:8.0.38]
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:687) [tomcat-coyote.jar:8.0.38]
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1520) [tomcat-coyote.jar:8.0.38]
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1476) [tomcat-coyote.jar:8.0.38]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_45]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_45]
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-util.jar:8.0.38]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
2016-12-12 15:29:42,783[io-8080-exec-14]  [INFO ] [o.e.c.a.w.s.WorkspaceManager 342]    - Workspace '66666666666666666666666' removed by user '777777777'
2016-12-12 15:29:42,875[io-8080-exec-23]  [INFO ] [o.e.c.a.w.s.WorkspaceManager 342]    - Workspace '88888888888888888' removed by user '999999999999'

Reproduction Steps:

Don't know, need to investigate.

Expected behavior:

No errors.

Observed behavior:

Errors in log.

Codenvy version: 5.0.0-M8
OS and version: Centos 7.2

Additional information:

Problem only started happening recently, didn't happen in an older version of Codenvy: Yes
Problem can be reliably reproduced: No - it's random

[skabashnyuk]

depends on eclipse-che/che#3258

[skabashnyuk]

We not sure that this issue is related to eclipse-che/che#3258
checking

[mkuznyetsov]

It seems that workspace has been deleted in two threads: last two notes in original logs about "Workspace 'X' removed by user 'Y' are actually about the same workspace. Hence, one tread successfully removed the snapshot, but another one failed to do so and threw an exception

[mkuznyetsov]

Because of that, it seems to be not a critical issue so far, as well as there are no ways of reproducing it locally now on M9. We decided to close it until there will be further occurrences in M9 release.