Skip to content

fix(alerts): Batch NotificationMessage delete metric alert rows#115726

Merged
ceorourke merged 2 commits into
masterfrom
ceorourke/batch-notificationmessage-deletes
May 18, 2026
Merged

fix(alerts): Batch NotificationMessage delete metric alert rows#115726
ceorourke merged 2 commits into
masterfrom
ceorourke/batch-notificationmessage-deletes

Conversation

@ceorourke
Copy link
Copy Markdown
Member

Batch the deletion here because the table is so large it ran for 12 hours and timed out.

@ceorourke ceorourke requested a review from a team as a code owner May 18, 2026 16:07
@github-actions github-actions Bot added the Scope: Backend Automatically applied to PRs that change backend components label May 18, 2026
Copy link
Copy Markdown
Contributor

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 4dca709. Configure here.

row.delete()
while True:
ids = list(
NotificationMessage.objects.filter(action__isnull=True).values_list("id", flat=True)[
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Filter criteria changed, may delete unintended rows

High Severity

The filter changed from incident__isnull=False, trigger_action__isnull=False (specifically metric alert rows) to action__isnull=True (any row without a workflow action). Despite the function being named delete_old_rows_metric_alerts, the new filter is strictly broader — it matches all rows lacking an action, including any orphaned rows where incident and trigger_action are also null. The PR description says this is just a batching change, but the filter criteria silently changed too, which could cause unintended data deletion.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 4dca709. Configure here.

@github-actions
Copy link
Copy Markdown
Contributor

Backend Test Failures

Failures on c60e08e in this run:

tests/sentry/sentry_apps/tasks/test_sentry_apps.py::TestDisabledSentryAppWebhooks::test_send_metric_alert_webhook_blocked_for_disabled_applog
[gw0] linux -- Python 3.13.1 /home/runner/work/sentry/sentry/.venv/bin/python3
src/sentry/testutils/cases.py:404: in _pre_setup
    super()._pre_setup()
src/sentry/utils/db.py:64: in _enter
    return original_enter(self)
           ^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.13/site-packages/django/db/transaction.py:211: in __enter__
    sid = connection.savepoint()
          ^^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.13/site-packages/django/utils/asyncio.py:26: in inner
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.13/site-packages/django/db/backends/base/base.py:400: in savepoint
    self._savepoint(sid)
.venv/lib/python3.13/site-packages/django/db/backends/base/base.py:367: in _savepoint
    cursor.execute(self.ops.savepoint_create_sql(sid))
.venv/lib/python3.13/site-packages/django/db/backends/utils.py:122: in execute
    return super().execute(sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.13/site-packages/sentry_sdk/utils.py:1887: in runner
    return original_function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.13/site-packages/django/db/backends/utils.py:79: in execute
    return self._execute_with_wrappers(
.venv/lib/python3.13/site-packages/django/db/backends/utils.py:92: in _execute_with_wrappers
    return executor(sql, params, many, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
src/sentry/db/postgres/base.py:70: in _execute__include_sql_in_error
    return execute(sql, params, many, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
src/sentry/db/postgres/base.py:58: in _execute__clean_params
    return execute(sql, clean_bad_params(params), many, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
src/sentry/testutils/hybrid_cloud.py:129: in __call__
    raise CrossTransactionAssertionError(
E   sentry.testutils.hybrid_cloud.CrossTransactionAssertionError: Transaction opened for db {'default'}, but command running against db control
E   SQL: SAVEPOINT "s140170946993024_x9515"
tests/sentry/sentry_apps/tasks/test_sentry_apps.py::TestDisabledSentryAppWebhooks::test_workflow_notification_blocked_for_disabled_applog
[gw0] linux -- Python 3.13.1 /home/runner/work/sentry/sentry/.venv/bin/python3
src/sentry/testutils/cases.py:404: in _pre_setup
    super()._pre_setup()
src/sentry/utils/db.py:64: in _enter
    return original_enter(self)
           ^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.13/site-packages/django/db/transaction.py:211: in __enter__
    sid = connection.savepoint()
          ^^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.13/site-packages/django/utils/asyncio.py:26: in inner
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.13/site-packages/django/db/backends/base/base.py:400: in savepoint
    self._savepoint(sid)
.venv/lib/python3.13/site-packages/django/db/backends/base/base.py:367: in _savepoint
    cursor.execute(self.ops.savepoint_create_sql(sid))
.venv/lib/python3.13/site-packages/django/db/backends/utils.py:122: in execute
    return super().execute(sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.13/site-packages/sentry_sdk/utils.py:1887: in runner
    return original_function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.13/site-packages/django/db/backends/utils.py:79: in execute
    return self._execute_with_wrappers(
.venv/lib/python3.13/site-packages/django/db/backends/utils.py:92: in _execute_with_wrappers
    return executor(sql, params, many, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
src/sentry/db/postgres/base.py:70: in _execute__include_sql_in_error
    return execute(sql, params, many, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
src/sentry/db/postgres/base.py:58: in _execute__clean_params
    return execute(sql, clean_bad_params(params), many, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
src/sentry/testutils/hybrid_cloud.py:129: in __call__
    raise CrossTransactionAssertionError(
E   sentry.testutils.hybrid_cloud.CrossTransactionAssertionError: Transaction opened for db {'default'}, but command running against db control
E   SQL: SAVEPOINT "s140170946993024_x9521"
tests/sentry/snuba/test_discover_timeseries_query.py::DiscoverTimeseriesQueryTest::test_comparison_aggregate_function_invalidlog
[gw0] linux -- Python 3.13.1 /home/runner/work/sentry/sentry/.venv/bin/python3
tests/sentry/snuba/test_discover_timeseries_query.py:32: in setUp
    project_id=self.project.id,
               ^^^^^^^^^^^^
.venv/lib/python3.13/site-packages/django/utils/functional.py:47: in __get__
    res = instance.__dict__[self.name] = self.func(instance)
                                         ^^^^^^^^^^^^^^^^^^^
src/sentry/testutils/fixtures.py:122: in project
    return self.create_project(name="Bar", slug="bar", teams=[self.team])
                                                              ^^^^^^^^^
.venv/lib/python3.13/site-packages/django/utils/functional.py:47: in __get__
    res = instance.__dict__[self.name] = self.func(instance)
                                         ^^^^^^^^^^^^^^^^^^^
/opt/hostedtoolcache/Python/3.13.1/x64/lib/python3.13/contextlib.py:85: in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
src/sentry/testutils/fixtures.py:113: in team
    team = self.create_team(organization=self.organization, name="foo", slug="foo")
                                         ^^^^^^^^^^^^^^^^^
.venv/lib/python3.13/site-packages/django/utils/functional.py:47: in __get__
    res = instance.__dict__[self.name] = self.func(instance)
                                         ^^^^^^^^^^^^^^^^^^^
src/sentry/testutils/fixtures.py:108: in organization
    return self.create_organization(name="baz", slug="baz", owner=self.user)
                                                                  ^^^^^^^^^
.venv/lib/python3.13/site-packages/django/utils/functional.py:47: in __get__
    res = instance.__dict__[self.name] = self.func(instance)
                                         ^^^^^^^^^^^^^^^^^^^
src/sentry/testutils/fixtures.py:94: in user
    return self.create_user(
src/sentry/testutils/fixtures.py:313: in create_user
    return Factories.create_user(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/opt/hostedtoolcache/Python/3.13.1/x64/lib/python3.13/contextlib.py:85: in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
src/sentry/testutils/factories.py:1102: in create_user
    user.save(is_test_user=is_test_user)
src/sentry/silo/base.py:157: in override
    return original_method(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
src/sentry/users/models/user.py:244: in save
    with outbox_context(transaction.atomic(using=router.db_for_write(User))):
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/opt/hostedtoolcache/Python/3.13.1/x64/lib/python3.13/contextlib.py:141: in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
src/sentry/hybridcloud/models/outbox.py:552: in outbox_context
    with unguarded_write(using=inner.using), enforce_constraints(inner):
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
... (26 more lines)
tests/sentry/snuba/test_errors.py::ErrorsQueryIntegrationTest::test_auto_fields_aggregateslog
[gw0] linux -- Python 3.13.1 /home/runner/work/sentry/sentry/.venv/bin/python3
src/sentry/testutils/cases.py:404: in _pre_setup
    super()._pre_setup()
src/sentry/utils/db.py:64: in _enter
    return original_enter(self)
           ^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.13/site-packages/django/db/transaction.py:211: in __enter__
    sid = connection.savepoint()
          ^^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.13/site-packages/django/utils/asyncio.py:26: in inner
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.13/site-packages/django/db/backends/base/base.py:400: in savepoint
    self._savepoint(sid)
.venv/lib/python3.13/site-packages/django/db/backends/base/base.py:367: in _savepoint
    cursor.execute(self.ops.savepoint_create_sql(sid))
.venv/lib/python3.13/site-packages/django/db/backends/utils.py:122: in execute
    return super().execute(sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.13/site-packages/sentry_sdk/utils.py:1887: in runner
    return original_function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.13/site-packages/django/db/backends/utils.py:79: in execute
    return self._execute_with_wrappers(
.venv/lib/python3.13/site-packages/django/db/backends/utils.py:92: in _execute_with_wrappers
    return executor(sql, params, many, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
src/sentry/db/postgres/base.py:70: in _execute__include_sql_in_error
    return execute(sql, params, many, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
src/sentry/db/postgres/base.py:58: in _execute__clean_params
    return execute(sql, clean_bad_params(params), many, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
src/sentry/testutils/hybrid_cloud.py:129: in __call__
    raise CrossTransactionAssertionError(
E   sentry.testutils.hybrid_cloud.CrossTransactionAssertionError: Transaction opened for db {'default'}, but command running against db control
E   SQL: SAVEPOINT "s140170946993024_x9544"
tests/sentry/snuba/test_errors.py::ErrorsQueryIntegrationTest::test_project_mappinglog
[gw0] linux -- Python 3.13.1 /home/runner/work/sentry/sentry/.venv/bin/python3
src/sentry/testutils/cases.py:404: in _pre_setup
    super()._pre_setup()
src/sentry/utils/db.py:64: in _enter
    return original_enter(self)
           ^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.13/site-packages/django/db/transaction.py:211: in __enter__
    sid = connection.savepoint()
          ^^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.13/site-packages/django/utils/asyncio.py:26: in inner
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.13/site-packages/django/db/backends/base/base.py:400: in savepoint
    self._savepoint(sid)
.venv/lib/python3.13/site-packages/django/db/backends/base/base.py:367: in _savepoint
    cursor.execute(self.ops.savepoint_create_sql(sid))
.venv/lib/python3.13/site-packages/django/db/backends/utils.py:122: in execute
    return super().execute(sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.13/site-packages/sentry_sdk/utils.py:1887: in runner
    return original_function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.13/site-packages/django/db/backends/utils.py:79: in execute
    return self._execute_with_wrappers(
.venv/lib/python3.13/site-packages/django/db/backends/utils.py:92: in _execute_with_wrappers
    return executor(sql, params, many, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
src/sentry/db/postgres/base.py:70: in _execute__include_sql_in_error
    return execute(sql, params, many, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
src/sentry/db/postgres/base.py:58: in _execute__clean_params
    return execute(sql, clean_bad_params(params), many, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
src/sentry/testutils/hybrid_cloud.py:129: in __call__
    raise CrossTransactionAssertionError(
E   sentry.testutils.hybrid_cloud.CrossTransactionAssertionError: Transaction opened for db {'default'}, but command running against db control
E   SQL: SAVEPOINT "s140170946993024_x9550"
tests/sentry/tasks/test_commits.py::FetchCommitsTest::test_github_compare_commits_cache_key_avoids_ambiguous_id_collisionslog
[gw0] linux -- Python 3.13.1 /home/runner/work/sentry/sentry/.venv/bin/python3
src/sentry/testutils/cases.py:411: in _post_teardown
    super()._post_teardown()
.venv/lib/python3.13/site-packages/django/db/backends/postgresql/base.py:482: in check_constraints
    cursor.execute("SET CONSTRAINTS ALL IMMEDIATE")
.venv/lib/python3.13/site-packages/django/db/backends/utils.py:122: in execute
    return super().execute(sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.13/site-packages/sentry_sdk/utils.py:1887: in runner
    return original_function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.13/site-packages/django/db/backends/utils.py:79: in execute
    return self._execute_with_wrappers(
.venv/lib/python3.13/site-packages/django/db/backends/utils.py:92: in _execute_with_wrappers
    return executor(sql, params, many, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
src/sentry/db/postgres/base.py:70: in _execute__include_sql_in_error
    return execute(sql, params, many, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
src/sentry/db/postgres/base.py:58: in _execute__clean_params
    return execute(sql, clean_bad_params(params), many, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
src/sentry/testutils/hybrid_cloud.py:129: in __call__
    raise CrossTransactionAssertionError(
E   sentry.testutils.hybrid_cloud.CrossTransactionAssertionError: Transaction opened for db {'secondary'}, but command running against db control
E   SQL: SET CONSTRAINTS ALL IMMEDIATE
tests/sentry/utils/test_query.py::RangeQuerySetWrapperTest::test_order_by_non_unique_failslog
[gw0] linux -- Python 3.13.1 /home/runner/work/sentry/sentry/.venv/bin/python3
src/sentry/testutils/cases.py:411: in _post_teardown
    super()._post_teardown()
.venv/lib/python3.13/site-packages/django/db/backends/postgresql/base.py:482: in check_constraints
    cursor.execute("SET CONSTRAINTS ALL IMMEDIATE")
.venv/lib/python3.13/site-packages/django/db/backends/utils.py:122: in execute
    return super().execute(sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.13/site-packages/sentry_sdk/utils.py:1887: in runner
    return original_function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.13/site-packages/django/db/backends/utils.py:79: in execute
    return self._execute_with_wrappers(
.venv/lib/python3.13/site-packages/django/db/backends/utils.py:92: in _execute_with_wrappers
    return executor(sql, params, many, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
src/sentry/db/postgres/base.py:70: in _execute__include_sql_in_error
    return execute(sql, params, many, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
src/sentry/db/postgres/base.py:58: in _execute__clean_params
    return execute(sql, clean_bad_params(params), many, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
src/sentry/testutils/hybrid_cloud.py:129: in __call__
    raise CrossTransactionAssertionError(
E   sentry.testutils.hybrid_cloud.CrossTransactionAssertionError: Transaction opened for db {'secondary'}, but command running against db control
E   SQL: SET CONSTRAINTS ALL IMMEDIATE
tests/sentry_plugins/test_client.py::ApiClientTest::test_postlog
[gw0] linux -- Python 3.13.1 /home/runner/work/sentry/sentry/.venv/bin/python3
src/sentry/testutils/cases.py:411: in _post_teardown
    super()._post_teardown()
.venv/lib/python3.13/site-packages/django/db/backends/postgresql/base.py:482: in check_constraints
    cursor.execute("SET CONSTRAINTS ALL IMMEDIATE")
.venv/lib/python3.13/site-packages/django/db/backends/utils.py:122: in execute
    return super().execute(sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.13/site-packages/sentry_sdk/utils.py:1887: in runner
    return original_function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.venv/lib/python3.13/site-packages/django/db/backends/utils.py:79: in execute
    return self._execute_with_wrappers(
.venv/lib/python3.13/site-packages/django/db/backends/utils.py:92: in _execute_with_wrappers
    return executor(sql, params, many, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
src/sentry/db/postgres/base.py:70: in _execute__include_sql_in_error
    return execute(sql, params, many, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
src/sentry/db/postgres/base.py:58: in _execute__clean_params
    return execute(sql, clean_bad_params(params), many, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
src/sentry/testutils/hybrid_cloud.py:129: in __call__
    raise CrossTransactionAssertionError(
E   sentry.testutils.hybrid_cloud.CrossTransactionAssertionError: Transaction opened for db {'secondary'}, but command running against db control
E   SQL: SET CONSTRAINTS ALL IMMEDIATE

@ceorourke ceorourke merged commit 5260013 into master May 18, 2026
117 of 119 checks passed
@ceorourke ceorourke deleted the ceorourke/batch-notificationmessage-deletes branch May 18, 2026 17:37
ceorourke added a commit that referenced this pull request May 19, 2026
…15578)

Follow up to #115529 to start
removing the `incident` and `trigger_action` column on
`NotifcationMessage`. Since only `action` and `group` will be used now,
I'm trying to make those not nullable.

Not to be merged until #115726
has been applied
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Scope: Backend Automatically applied to PRs that change backend components

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants