fix(deletions): Prevent timeouts when deleting GroupHash records #101720

armenzg · 2025-10-17T13:55:45Z

The issues comes from this block:

sentry/src/sentry/deletions/defaults/group.py

Lines 248 to 259 in a3a7717

    
           try: 
        
               if seer_deletion: 
        
                   # Tell seer to delete grouping records for these groups 
        
                   # It's low priority to delete the hashes from seer, so we don't want 
        
                   # any network errors to block the deletion of the groups 
        
                   hash_values = [gh[1] for gh in hashes_chunk] 
        
                   may_schedule_task_to_delete_hashes_from_seer(project_id, hash_values) 
        
           except Exception: 
        
               logger.warning("Error scheduling task to delete hashes from seer") 
        
           finally: 
        
               hash_ids = [gh[0] for gh in hashes_chunk] 
        
               GroupHash.objects.filter(id__in=hash_ids).delete()

The update is triggered because of this on_delete:

sentry/src/sentry/models/grouphashmetadata.py

Lines 116 to 118 in b1f684a

    
           seer_matched_grouphash = FlexibleForeignKey( 
        
               "sentry.GroupHash", related_name="seer_matchees", on_delete=models.SET_NULL, null=True 
        
           )

Currently, when we try to delete all the group hashes, we update the related group hash metadata first. This query ends up failing for taking longer than 30 seconds:

SQL: UPDATE "sentry_grouphashmetadata" SET "seer_matched_grouphash_id" = NULL WHERE "sentry_grouphashmetadata"."seer_matched_grouphash_id" IN (%s, ..., %s)

This can be resolved by deleting the group hash metadata rows before trying to delete the group hash rows. This will avoid the update statement altogether.

This fix was initially started in #101545, however, the solution has completely changed, thus, starting a new PR.

Fixes SENTRY-5ABJ.

The issues comes from this block: https://github.com/getsentry/sentry/blob/a3a771719d4777bd747d98fb05eb77c20425e3d6/src/sentry/deletions/defaults/group.py#L248-L259 The update is triggered because of this `on_delete`: https://github.com/getsentry/sentry/blob/b1f684a335128dbc74ad3a7fac1d7052df9e8f01/src/sentry/models/grouphashmetadata.py#L116-L118 Currently, when we try to delete all the group hashes, we update the related group hash metadata first. This query ends up failing for taking longer than 30 seconds: > SQL: UPDATE "sentry_grouphashmetadata" SET "seer_matched_grouphash_id" = NULL WHERE "sentry_grouphashmetadata"."seer_matched_grouphash_id" IN (%s, ..., %s) This can be resolved by deleting the group hash _metadata_ rows before trying to delete the group hash rows. This will avoid the update statement altogether. This fix was initially started in #101545, however, the solution has completely changed, thus, starting a new PR. Fixes [SENTRY-5ABJ](https://sentry.io/organizations/sentry/issues/6930113529/).

armenzg · 2025-10-17T13:56:26Z

src/sentry/deletions/defaults/group.py


        iterations += 1

+    if iterations == GROUP_HASH_ITERATIONS:


Drive-by metric.

armenzg · 2025-10-17T13:57:26Z

src/sentry/deletions/defaults/group.py

+                    GroupHashMetadata.objects.filter(grouphash_id__in=hash_ids).delete()
+                except Exception:
+                    # XXX: Let's make sure that no issues are caused by this and then remove it
+                    logger.exception("Error deleting group hash metadata")


Once I enable the option, I would like to know if any problems are caused by this while falling back to the original behaviour rather than completely aborting the process.

armenzg · 2025-10-17T13:57:49Z

src/sentry/models/grouphash.py


-    __repr__ = sane_repr("group_id", "hash")
+    __repr__ = sane_repr("group_id", "hash", "metadata")
+    __str__ = __repr__


It makes print statements during debugging actually useful.

armenzg · 2025-10-17T13:58:23Z

src/sentry/options/defaults.py

 register(
    "deletions.group-hashes-batch-size",
-    default=10000,
+    default=100,


It's already 100 in options:
https://github.com/getsentry/sentry-options-automator/blob/d684f59a4318132595dd3529f3fc4d32467b364f/options/default/app.yaml#L256

armenzg · 2025-10-17T13:58:34Z

src/sentry/options/defaults.py

+    flags=FLAG_AUTOMATOR_MODIFIABLE,
+)
+register(
+    "deletions.group.delete_group_hashes_metadata_first",


This will control the new behaviour.

armenzg · 2025-10-17T13:59:35Z

tests/sentry/deletions/test_group.py

+            assert grouphash_a.metadata is not None
+            assert grouphash_a.metadata.seer_matched_grouphash is None
+            assert grouphash_b.metadata is not None
+            assert grouphash_b.metadata.seer_matched_grouphash == grouphash_a


This verifies that we have the column set to the first group hash. Not that we end up doing the update statement, however, I want to verify that we're testing the same code path.

armenzg · 2025-10-17T14:06:14Z

tests/sentry/deletions/test_group.py

+        """
+        Test that when deleting group hashes, the group hash metadata is deleted first (which will not update the references to the other group hashes)
+        """
+        with self.options({"deletions.group.delete_group_hashes_metadata_first": True}):


The two tests are functionally the same.
This test avoids the update call.

armenzg · 2025-10-17T14:11:45Z

bugbot run

armenzg · 2025-10-17T14:13:19Z

src/sentry/deletions/defaults/group.py

+                # gh B -> ghm B -> gh C
+                # gh C -> ghm C -> gh A
+                #
+                # Deleting group hashes A, B & C (since they all point to the same group) will require:


This function is called as this: delete_group_hashes(project_id, group_ids), thus, all group hashes point to the same group.

armenzg · 2025-10-17T14:14:49Z

src/sentry/deletions/defaults/group.py

+                # Deleting group hashes A, B & C (since they all point to the same group) will require:
+                # * Updating columns ghmB & ghmC to point to None
+                # * Deleting the group hash metadata rows
+                # * Deleting the group hashes


Before this PR, the current approach is this:

Start group hashes deletion

Which triggers group hash metadata column updating

Which then triggers deleting the group hash metadata rows

Now that children group hash metadata are deleted we can delete the group hashes

codecov · 2025-10-17T14:26:35Z

Codecov Report

❌ Patch coverage is 73.33333% with 4 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
src/sentry/deletions/defaults/group.py	60.00%	4 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           master   #101720      +/-   ##
===========================================
+ Coverage   80.85%    80.95%   +0.09%     
===========================================
  Files        8707      8707              
  Lines      387091    387104      +13     
  Branches    24524     24524              
===========================================
+ Hits       313000    313393     +393     
+ Misses      73743     73363     -380     
  Partials      348       348

markstory · 2025-10-17T14:37:20Z

tests/sentry/deletions/test_group.py

+            # Verify that seer matched event_b to event_a's hash
+            assert event_a.group_id == event_b.group_id
+            # Make sure it has not changed
+            assert grouphash_a.metadata is not None


The django ORM record won't change even if the underlying db record was changed. You could add grouphash_a.refresh_from_db() to ensure that you have the latest data from postgres.

I forgot to push that change. I have it locally. I will make it part of my next PR.

I added this in #101796

sentry-io · 2025-10-17T14:53:56Z

Issues attributed to commits in this pull request

This pull request was merged and Sentry observed the following issues:

‼️ ThreadLeakAssertionError: <Thread(Thread-7 (run),... in local

armenzg · 2025-10-17T15:07:51Z

Issues attributed to commits in this pull request

This pull request was merged and Sentry observed the following issues:

‼️ ThreadLeakAssertionError: <Thread(Thread-7 (run),... in local

This is part of the ThreadLeaks project and not part of the CI test runs. I can see my host name in there.

Last week when I was debugging tests for #101720 it was confusing to find events and groups that had nothing to do with the tests I was working on. This refactor moves the majority of the logic from the setUp function to the first test since it's where its needed.

This was added in #101720.

This was added in #101720 and it's not needed anymore.

armenzg self-assigned this Oct 17, 2025

github-actions bot added the Scope: Backend Automatically applied to PRs that change backend components label Oct 17, 2025

vercel bot deployed to Preview October 17, 2025 13:57 View deployment

This comment was marked as outdated.

Sign in to view

Add extra test

e21a73e

vercel bot deployed to Preview October 17, 2025 14:07 View deployment

Minor change

0518301

vercel bot deployed to Preview October 17, 2025 14:10 View deployment

armenzg commented Oct 17, 2025

View reviewed changes

This comment was marked as outdated.

Sign in to view

armenzg commented Oct 17, 2025

View reviewed changes

armenzg marked this pull request as ready for review October 17, 2025 14:15

armenzg requested review from a team as code owners October 17, 2025 14:15

armenzg requested a review from markstory October 17, 2025 14:15

markstory approved these changes Oct 17, 2025

View reviewed changes

armenzg merged commit 03e49a0 into master Oct 17, 2025
69 checks passed

armenzg deleted the 0/delete_group_hashes_metadata_first/armenzg branch October 17, 2025 14:48

armenzg mentioned this pull request Oct 20, 2025

ref(tests): Remove setUp function #101796

Merged

armenzg added a commit that referenced this pull request Oct 22, 2025

chore(cleanup): Drop option for deleting group hash metadata

c19d58f

This was added in #101720.

armenzg mentioned this pull request Oct 22, 2025

chore(cleanup): Drop option for deleting group hash metadata #101918

Merged

armenzg added a commit that referenced this pull request Oct 23, 2025

chore(cleanup): Drop option for deleting group hash metadata (#101918)

f4f0faf

This was added in #101720 and it's not needed anymore.

	try:
	if seer_deletion:
	# Tell seer to delete grouping records for these groups
	# It's low priority to delete the hashes from seer, so we don't want
	# any network errors to block the deletion of the groups
	hash_values = [gh[1] for gh in hashes_chunk]
	may_schedule_task_to_delete_hashes_from_seer(project_id, hash_values)
	except Exception:
	logger.warning("Error scheduling task to delete hashes from seer")
	finally:
	hash_ids = [gh[0] for gh in hashes_chunk]
	GroupHash.objects.filter(id__in=hash_ids).delete()

	seer_matched_grouphash = FlexibleForeignKey(
	"sentry.GroupHash", related_name="seer_matchees", on_delete=models.SET_NULL, null=True
	)

Uh oh!

fix(deletions): Prevent timeouts when deleting GroupHash records #101720

fix(deletions): Prevent timeouts when deleting GroupHash records #101720

Conversation

armenzg commented Oct 17, 2025

Uh oh!

This comment was marked as outdated.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

armenzg commented Oct 17, 2025

Uh oh!

This comment was marked as outdated.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov bot commented Oct 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sentry-io bot commented Oct 17, 2025

Issues attributed to commits in this pull request

Uh oh!

armenzg commented Oct 17, 2025

Issues attributed to commits in this pull request

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codecov bot commented Oct 17, 2025 •

edited

Loading