feat(deletions): Schedule task to delete pending deletions groups #103820

armenzg · 2025-11-21T16:30:49Z

Sometimes, groups do not get deleted and end up in limbo.

This task will handle groups that are pending deletion before 90 days retention and older than a day.

Sometimes, groups do not get deleted and end up in limbo. This task will handle groups that are pending deletion before 90 days retention and older than a day.

armenzg · 2025-11-21T16:31:47Z

src/sentry/conf/server.py

+        "task": "deletions:sentry.tasks.delete_pending_groups",
+        # Runs every 2 hours during 9am-5pm Eastern Time (EST: UTC-5)
+        # 9am, 11am, 1pm, 3pm, 5pm EST = 14:00, 16:00, 18:00, 20:00, 22:00 UTC
+        "schedule": task_crontab("0", "14,16,18,20,22", "*", "*", "*"),


The schedule now is wonky. I want to see it run during my working hours. Once there are no issues I will switch it to once a day.

armenzg · 2025-11-21T16:32:23Z

src/sentry/tasks/delete_pending_groups.py

+    # This can timeout for lack of an index on the status field
+    # Not using the last_seen index to avoid the lack of composite index on status and last_seen
+    groups = Group.objects.filter(status__in=statuses_to_delete).values_list(
+        "id", "project_id", "last_seen"


Grabbing last_seen here to filter it out programatically down below.

armenzg · 2025-11-21T16:33:11Z

src/sentry/tasks/delete_pending_groups.py

+    # Not using the last_seen index to avoid the lack of composite index on status and last_seen
+    groups = Group.objects.filter(status__in=statuses_to_delete).values_list(
+        "id", "project_id", "last_seen"
+    )[:BATCH_LIMIT]


Since there's no index on status, let's grab a few a time. Currently, the query takes about 20 seconds.

We have a compund index with status being the first on the index so it should be hitting and using it, you can try and EXPLAIN the query and you will see its hitting the index

sentry/src/sentry/models/group.py

Lines 635 to 643 in b2fff1d

status = BoundedPositiveIntegerField(

default=GroupStatus.UNRESOLVED,

choices=(

(GroupStatus.UNRESOLVED, _("Unresolved")),

(GroupStatus.RESOLVED, _("Resolved")),

(GroupStatus.IGNORED, _("Ignored")),

),

db_index=True,

)

It looks like there is an index on status, and checking locally I have an index on the status column as well.

Oh my! I'm glad it's there. I may have been looking at the list of compound indexes.

armenzg · 2025-11-21T16:33:52Z

src/sentry/tasks/delete_pending_groups.py

+    )
+
+    for project_id, group_ids in groups_by_project.items():
+        # Schedule deletion tasks in chunks of GROUP_CHUNK_SIZE


Similar to how we schedule it here:

sentry/src/sentry/api/helpers/group_index/delete.py

Lines 87 to 95 in 1483996

# Schedule a task per GROUP_CHUNK_SIZE batch of groups

for i in range(0, len(group_ids), GROUP_CHUNK_SIZE):

delete_groups_for_project.apply_async(

kwargs={

"project_id": project.id,

"object_ids": group_ids[i : i + GROUP_CHUNK_SIZE],

"transaction_id": str(transaction_id),

}

)

armenzg · 2025-11-21T16:34:55Z

src/sentry/tasks/delete_pending_groups.py

+            chunk = group_ids[i : i + GROUP_CHUNK_SIZE]
+            transaction_id = str(uuid4())
+
+            delete_groups_for_project.apply_async(


As reference, the called task:

sentry/src/sentry/deletions/tasks/groups.py

Line 25 in 1483996

def delete_groups_for_project(

codecov · 2025-11-21T16:51:30Z

Codecov Report

❌ Patch coverage is 95.34884% with 2 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
src/sentry/tasks/delete_pending_groups.py	95.34%	2 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff            @@
##           master   #103820   +/-   ##
========================================
  Coverage   80.62%    80.62%           
========================================
  Files        9281      9282    +1     
  Lines      396223    396285   +62     
  Branches    25256     25256           
========================================
+ Hits       319438    319488   +50     
- Misses      76325     76337   +12     
  Partials      460       460

yuvmen

looks good! one minor comment

yuvmen · 2025-11-21T16:50:58Z

src/sentry/tasks/delete_pending_groups.py

+    # Not using the last_seen index to avoid the lack of composite index on status and last_seen
+    groups = Group.objects.filter(status__in=statuses_to_delete).values_list(
+        "id", "project_id", "last_seen"
+    )[:BATCH_LIMIT]


We have a compund index with status being the first on the index so it should be hitting and using it, you can try and EXPLAIN the query and you will see its hitting the index

yuvmen · 2025-11-21T16:52:04Z

src/sentry/tasks/delete_pending_groups.py

+
+    for project_id, group_ids in groups_by_project.items():
+        # Schedule deletion tasks in chunks of GROUP_CHUNK_SIZE
+        for i in range(0, len(group_ids), GROUP_CHUNK_SIZE):


nit, we got the chunked function you can use

I will handle this later. I want to merge it now.

markstory

Makes sense to me. These scheduled tasks will help cleanup any API based deletions that failed and aren't being retried as there isn't a scheduled deletion row to restart those deletions currently.

markstory · 2025-11-21T16:58:17Z

src/sentry/tasks/delete_pending_groups.py

+    # Not using the last_seen index to avoid the lack of composite index on status and last_seen
+    groups = Group.objects.filter(status__in=statuses_to_delete).values_list(
+        "id", "project_id", "last_seen"
+    )[:BATCH_LIMIT]


sentry/src/sentry/models/group.py

Lines 635 to 643 in b2fff1d

status = BoundedPositiveIntegerField(

default=GroupStatus.UNRESOLVED,

choices=(

(GroupStatus.UNRESOLVED, _("Unresolved")),

(GroupStatus.RESOLVED, _("Resolved")),

(GroupStatus.IGNORED, _("Ignored")),

),

db_index=True,

)

It looks like there is an index on status, and checking locally I have an index on the status column as well.

This is a follow-up to #103820. A failure to delete a group should be re-processed within 24 hours. Changes included: * Schedule re-processing task every 6 hours * Re-process groups pending deletion that are older than 6 hours rather than midnight from the day before

This is a follow-up to #103820. A failure to delete a group should be re-processed within 24 hours. Changes included: * Schedule re-processing task every N hours instead of odd schedule * Re-process groups pending deletion that are older than 6 hours rather than midnight from the day before

feat(deletions): Schedule task to delete pending deletions groups

939c2e8

Sometimes, groups do not get deleted and end up in limbo. This task will handle groups that are pending deletion before 90 days retention and older than a day.

armenzg self-assigned this Nov 21, 2025

github-actions bot added the Scope: Backend Automatically applied to PRs that change backend components label Nov 21, 2025

vercel bot deployed to Preview November 21, 2025 16:33 View deployment

armenzg commented Nov 21, 2025

View reviewed changes

armenzg marked this pull request as ready for review November 21, 2025 16:35

armenzg requested review from markstory, wedamija and yuvmen November 21, 2025 16:35

Fix typing

e8a04f8

vercel bot deployed to Preview November 21, 2025 16:41 View deployment

yuvmen approved these changes Nov 21, 2025

View reviewed changes

markstory approved these changes Nov 21, 2025

View reviewed changes

armenzg merged commit b97b6a0 into master Nov 21, 2025
67 checks passed

armenzg deleted the 11_21/pending_groups/armenzg branch November 21, 2025 17:44

armenzg mentioned this pull request Nov 25, 2025

chore(deletions): Change schedule & date ranges #103897

Merged

	status = BoundedPositiveIntegerField(
	default=GroupStatus.UNRESOLVED,
	choices=(
	(GroupStatus.UNRESOLVED, _("Unresolved")),
	(GroupStatus.RESOLVED, _("Resolved")),
	(GroupStatus.IGNORED, _("Ignored")),
	),
	db_index=True,
	)

	# Schedule a task per GROUP_CHUNK_SIZE batch of groups
	for i in range(0, len(group_ids), GROUP_CHUNK_SIZE):
	delete_groups_for_project.apply_async(
	kwargs={
	"project_id": project.id,
	"object_ids": group_ids[i : i + GROUP_CHUNK_SIZE],
	"transaction_id": str(transaction_id),
	}
	)

Uh oh!

feat(deletions): Schedule task to delete pending deletions groups #103820

feat(deletions): Schedule task to delete pending deletions groups #103820

Conversation

armenzg commented Nov 21, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov bot commented Nov 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

yuvmen left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

markstory left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov bot commented Nov 21, 2025 •

edited

Loading