Skip to content

Account and workspace deletion error for large data volumes #391

@kjlippold

Description

@kjlippold

Steps to Reproduce

  1. Create a workspace with datastreams totaling about 1 million observations.
  2. Attempt to delete the account or workspace.

Expected Behavior

The account/workspace should finish deleting after a short loading period.

Actual Behavior

The delete form appears to hang, and the data will sometimes not be deleted due to the operation timing out.

Any additional comments?

There are a few things that need to happen to resolve this issue.

  1. The backend's CASCADE logic for observations needs to be further improved. We're not using Django's inefficient built-in cascade functionality, and this isn't an n+1 query issue. However, for millions of observations, our current method of filtering and deleting observations is still too slow and uses too much memory. We should explore setting up a PostgreSQL cascade delete for observations to see if that helps speed the operation up. After that, our best option would be to partition the observations table, which we've already done in the past and probably don't want to revisit.
  2. We could defer account and workspace deletion to a backend worker. This would allow the delete operation to take more time if needed without requiring the user to wait. We'd have to be able to mark accounts and workspaces as scheduled for deletion and make sure they can't be queried through the API. This is only an option if Celery is enabled on the instance.
  3. The frontend doesn't have loading indicators on account or workspace delete forms, so it appears the operation is hanging when the user selects it. We should add a loading indicator and grey out the delete button on delete forms when a user tries to delete a workspace or their account.

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions