Recurring/intermittent issue - user account env reset fails (configmaps fabric8-environments not found)

[ldimaggi]

Resetting a user's environment sometimes fails with this error:

Failed to load resource: the server responded with a status of 500 (Internal Server Error)
_cleanup:1 Form submission canceled because the form is not connected
api.openshift.io/api/spaces...
Failed to load resource: the server responded with a status of 500 (Internal Server Error)

Over time, I have seen this error less than 5% of the time - in debugging tests today, I have been seeing it ~20% of the time - randomly.

[joshuawilson]

I just ran into this. I ran the reset again and it cleared it out.

[ldimaggi]

For the most part, a retry resolves this problem. However, I have seem instances where multiple retries are needed. No details other than the 500 error are logged.

[joshuawilson]

I think this is a duplicate of #2867

[ldimaggi]

Seeing some additional patterns.

Before the reset operation that is to fail is started, the reset env page only displays a subset of the user's current spaces - and the formatting of the horizontal row for the spaces is truncated - in the following example, there were actually 2 spaces in the user's account:

Here's the correct set of spaces:

So - perhaps the error does not occur in the actual deletion of the spaces, but in the collecting of the spaces before the reset operation is performed?

[ldimaggi]

Seeing a pattern where additional information is available:

The 500/server error returned includes:

configmaps fabric8-environments not found

[ldimaggi]

This is happening repeatedly - this should be a SEV2.

[joshuawilson]

could be related to #3556

[ldimaggi]

+1 - that is the same error message as in #3556

[joshuawilson]

@aslakknutsen do you have any info or ideas on this?

[ebaron]

I think what is happening (with some help from @jiekang), is that the frontend is calling the Delete Space API and Clean Tenant API asynchronously [1]. Before these two APIs operated on different backend resources, but this is no longer true now that Delete Space also cleans up OpenShift resources. Interleaving these two API calls could trigger a variety of errors.

Perhaps we should make cleaning up OpenShift resources optional in the Delete Space API. Then it could do so when deleting individual spaces, but defer to the more robust/faster cleanup in Clean Tenant when resetting the environment.

[1] https://github.com/fabric8-ui/fabric8-ui/blob/7494b283e1b86875aae6592b119ace9c86dd2d3c/src/app/profile/cleanup/cleanup.component.ts#L90

[jiekang]

@ebaron I wouldn't mind fixing the race condition in cleanup.component.ts. What's the suggested API to use now? Is having it run the delete spaces one by one, followed by clean tenant, an okay fix?

[ebaron]

@jiekang Yes, I think what you described would be the way to fix it in the frontend. We could also add an optional parameter to the API where you can skip deleting OpenShift resources when deleting a space. Then I don't believe we would have to synchronize between deleting spaces and cleaning the tenant.

[jiekang]

Ah; so the longer path is to provide a parameter in the API, and then have the UI use it. Hmm...

[jiekang]

@ebaron Were you planning to take a look at providing that parameter?

[ebaron]

@jiekang Sure, if there are no objections, I can add this parameter to the Delete Space API. The frontend will have to make sure that this argument is set to true only when resetting the environment.

[jiekang]

I've assigned myself here as well. I can alter the frontend cleanup code to set the parameter to true.

[ldimaggi]

Just to confirm - the fix will be to delete spaces one by one, followed by clean tenant - correct?

[jiekang]

@ldimaggi We're currently going with a different fix:

Make delete space API have parameter that, when true, makes delete API not clash with clean tenant API. Then front-end is free to request delete spaces and clean tenant APIs at the same time.

[jiekang]

However I'm open to any opinions otherwise.

[ebaron]

I am open to suggestions as well. The clean tenant API clears a superset of what is cleared by the delete space API. When resetting the environment, this makes any OpenShift cleanup done by the delete space API redundant and also prone to racing with the clean tenant API.

[jiekang]

I'd prefer a single endpoint exist for the 'reset environment' action that handles it however it wants, but that might be exposing too much at a single point. If there aren't any opinions I think we can just go ahead with the optional parameter until someone objects :)

[ebaron]

@joshuawilson @jiekang Just a heads up, I opened a PR for the backend portion of the fix I proposed above (#3500 (comment)): fabric8-services/fabric8-wit#2121

[ebaron]

@jiekang there is now a "skipCluster" boolean argument for the delete space API in production. By setting this to true (e.g. DELETE https://openshift.io/api/spaces/780dc9bb-0f04-4e39-97cf-c3110be78005?skipCluster=true), WIT will not attempt to delete any OpenShift resources. We should specify this only when calling the delete space API when resetting the user's environment, since tenant will clean it up, and not when deleting an individual space.

[jiekang]

Okay I will look at opening a PR for that.

[ebaron]

All PRs have been merged to make use of the "skipCluster" argument when deleting spaces during an environment reset, and are now in production. If the issue reoccurs, feel free to reopen.