Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failure to delete large project #1002

Open
pombredanne opened this issue Nov 4, 2023 · 3 comments · May be fixed by #1060
Open

Failure to delete large project #1002

pombredanne opened this issue Nov 4, 2023 · 3 comments · May be fixed by #1060

Comments

@pombredanne
Copy link
Member

I have a project with about 1M files and 2GB compressed inputs (~10G uncompressed)
Deleting failed now and then with a "Gateway timeout" error

@pombredanne
Copy link
Member Author

This happens also when deleting many projects. "504 Gateway Time-out
nginx/1.19.6"

I would suggest that the deletion should happen in two steps:

  1. the project is marked for deletion and the UI returns immediately. The project is hidden from the UI
  2. there is a background task spawned for each project to delete. If it succeeds, the project record is deleted. If it fails, the project deletion marker is reset and the projects errors are updated with the project errors (or a banner is display with the error, like when a project has no longer its backing files)

@pombredanne
Copy link
Member Author

Note that things work fine from the command line.

@jayanth-kumar-morem
Copy link
Contributor

Working on this

jayanth-kumar-morem added a commit to jayanth-kumar-morem/scancode.io that referenced this issue Jan 26, 2024
1. Fix Default Filtering in ProjectFilterSet:
   - Modified the `ProjectFilterSet` to include `is_marked_for_deletion` in the default filtering when `is_archived` is not provided.
   - This ensures that projects marked for deletion are excluded from default views.

2. Migration for `is_marked_for_deletion`:
   - Added a migration (`0052_project_is_marked_for_deletion.py`) to introduce the `is_marked_for_deletion` field in the `Project` model.

3. Updated `Project` Model:
   - Added `is_marked_for_deletion` field to the `Project` model with a default value of `False`.

4. Modified `ProjectListView` to Exclude Marked Projects:
   - Modified the `get_queryset` method in `ProjectListView` to only fetch projects with `is_marked_for_deletion=False`.

5. Introduced `mark_for_deletion` Method:
   - Added a `mark_for_deletion` method in the `Project` model to set the `is_marked_for_deletion` flag and save the model.

6. Changed `delete` Method in `Project` Model:
   - Renamed the `delete` method to `delete_action` in the `Project` model to avoid conflicts.
   - Introduced a new `delete` method that marks the project for deletion and enqueues a background deletion task.

7. Background Deletion Task:
   - Created a background deletion task using `django_rq` to handle project deletion asynchronously.
   - Checks if the project is still marked for deletion before proceeding.
   - If an error occurs during deletion, updates project status and logs an error.

8. Updated `ProjectActionView` Success Message:
   - Modified the success message in `ProjectActionView` for the "delete" action to provide a more informative message based on the count.

Fixes: nexB#1002
Signed-off-by: Jayanth Kumar <jknani111@gmail.com>
jayanth-kumar-morem added a commit to jayanth-kumar-morem/scancode.io that referenced this issue Jan 27, 2024
1. Fix Default Filtering in ProjectFilterSet:
   - Modified the `ProjectFilterSet` to include `is_marked_for_deletion` in the default filtering when `is_archived` is not provided.
   - This ensures that projects marked for deletion are excluded from default views.

2. Migration for `is_marked_for_deletion`:
   - Added a migration (`0052_project_is_marked_for_deletion.py`) to introduce the `is_marked_for_deletion` field in the `Project` model.

3. Updated `Project` Model:
   - Added `is_marked_for_deletion` field to the `Project` model with a default value of `False`.

4. Modified `ProjectListView` to Exclude Marked Projects:
   - Modified the `get_queryset` method in `ProjectListView` to only fetch projects with `is_marked_for_deletion=False`.

5. Introduced `mark_for_deletion` Method:
   - Added a `mark_for_deletion` method in the `Project` model to set the `is_marked_for_deletion` flag and save the model.

6. Add `delete_in_background` Method in `Project` Model:
   - Introduced a new `delete_in_background` method that marks the project for deletion and enqueues a background deletion task.

7. Background Deletion Task:
   - Created a background deletion task using `django_rq` to handle project deletion asynchronously.
   - Checks if the project is still marked for deletion before proceeding.
   - If an error occurs during deletion, updates project status and logs an error.

8. Updated `ProjectActionView` Success Message:
   - Modified the success message in `ProjectActionView` for the "delete" action to provide a more informative message based on the count.

Fixes: nexB#1002
Signed-off-by: Jayanth Kumar <jknani111@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants