feat(deletion): Add partition support to BulkDeleteQuery and cleanup command#107906
Merged
feat(deletion): Add partition support to BulkDeleteQuery and cleanup command#107906
Conversation
Contributor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
Bugbot Autofix is OFF. To automatically fix reported issues with Cloud Agents, enable Autofix in the Cursor dashboard.
Add a `partition` parameter to BulkDeleteQuery that enables splitting
deletion work across multiple runs using modulo-based row bucketing.
When `partition=(bucket, total, key_column)` is provided, the DELETE
query adds `WHERE {key} % {total} = {bucket}`, so each run only handles
a fraction of eligible rows. This allows spreading deletion load across
multiple scheduled jobs to reduce dead tuple bursts and autovacuum
contention on high-churn tables like accounts_spike_projections.
The partition filter is applied in the inner SELECT subquery (where the
LIMIT is), ensuring each partition independently selects and deletes its
own candidate rows.
c6dba4c to
2e2f9c0
Compare
ajay-sentry
reviewed
Feb 10, 2026
| # Parse and validate --partition flag | ||
| parsed_partition: tuple[int, int, str] | None = None | ||
| if partition is not None: | ||
| parts = partition.split("/") |
Contributor
There was a problem hiding this comment.
a thought here, is we could have had 2 separate params here for total_buckets and bucket_id or smth to make it a little more straightforward
ajay-sentry
approved these changes
Feb 10, 2026
Contributor
ajay-sentry
left a comment
There was a problem hiding this comment.
looks good, I see partition_key isn't used on the cron but makes sense if we want to easily update later
…cleanup command
Expose BulkDeleteQuery's partition support via the `sentry cleanup` CLI:
--partition-bucket BUCKET (0-based bucket index)
--partition-total TOTAL (total number of buckets)
--partition-key COLUMN (default: id)
This allows K8s CronJobs to split bulk deletion work across multiple
scheduled runs. For example, the spikeprotections cleanup can be split
into 4 jobs at 6-hour intervals, each handling ~25% of eligible rows
via `id % 4 = {0,1,2,3}`.
Includes input validation: both flags must be used together, bucket
must be non-negative and less than total, total must be positive.
2e2f9c0 to
4191754
Compare
jaydgoss
pushed a commit
that referenced
this pull request
Feb 12, 2026
…command (#107906) ## Summary - Add `partition` parameter to `BulkDeleteQuery` to split deletion work across multiple runs using modulo-based row bucketing - Add `--partition-bucket`, `--partition-total`, and `--partition-key` flags to the `sentry cleanup` CLI command - Fully backward compatible: behavior is unchanged when partition flags are not provided ## Problem The daily `sentry cleanup` CronJob for `SpikeProjections` and `Spike` models runs a tight DELETE loop via `BulkDeleteQuery._continuous_query()`, creating a burst of dead tuples on `db-usage-1`. This triggers a massive autovacuum that causes WAL replication delay, forcing `db-usage-repl` to fall back to GSCP recovery. Since `valid_date` values are always at midnight UTC, simply running the CronJob more frequently doesn't help — all eligible rows become deletable at the same instant. We need a way to **partition the rows** across multiple runs. ## Solution Add `id % N` partitioning to `BulkDeleteQuery`: ``` sentry cleanup --model=SpikeProjections --days=90 --partition-bucket=0 --partition-total=4 sentry cleanup --model=SpikeProjections --days=90 --partition-bucket=1 --partition-total=4 sentry cleanup --model=SpikeProjections --days=90 --partition-bucket=2 --partition-total=4 sentry cleanup --model=SpikeProjections --days=90 --partition-bucket=3 --partition-total=4 ``` Each run adds `WHERE id % 4 = {bucket}` to the DELETE query, handling ~25% of eligible rows. The `--partition-key` flag allows using a different column (defaults to `id`). Using `id` (auto-increment) ensures uniform distribution, following the same principle as [PR #18736](getsentry/getsentry#18736) which switched spike projection batching from `organization_id` (snowflake, uneven) to `subscription.id` (auto-increment, uniform). ## Changes ### `src/sentry/db/deletion.py` - Added `partition: tuple[int, int, str] | None` parameter to `BulkDeleteQuery.__init__()` - Added partition filter to the WHERE clause in `execute()` - Added partition filter to `iterator()` via `Func(F(key), Value(total), function="MOD")` ### `src/sentry/runner/commands/cleanup.py` - Added `--partition-bucket` CLI flag (integer, 0-based bucket index) - Added `--partition-total` CLI flag (integer, total number of buckets) - Added `--partition-key` CLI flag (default: `id`) - Validation: both bucket and total must be used together, bucket must be non-negative and less than total, total must be positive - Threaded through `cleanup()` → `_cleanup()` → `run_bulk_query_deletes()` → `BulkDeleteQuery()` ## Test plan - [x] `test_partition_restriction` — verifies only rows in the matching bucket are deleted - [x] `test_partition_with_datetime_restriction` — combines partition + date filter - [x] `test_partition_all_buckets_cover_all_rows` — verifies complete coverage across all buckets - [x] `test_iteration_with_partition` — verifies `iterator()` respects partition filter - [x] `test_partition_bucket_exceeds_total` — validation error for bucket >= total - [x] `test_partition_negative_bucket` — validation error for negative bucket - [x] `test_partition_zero_total` — validation error for zero total - [x] `test_partition_bucket_without_total` — validation error when only bucket is set - [x] `test_partition_total_without_bucket` — validation error when only total is set ## Related - Ops PR (K8s config): getsentry/ops#19081 - Analysis: daily mass deletion on `accounts_spike_projections` causes replication delay on `db-usage-1`
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
partitionparameter toBulkDeleteQueryto split deletion work across multiple runs using modulo-based row bucketing--partition-bucket,--partition-total, and--partition-keyflags to thesentry cleanupCLI commandProblem
The daily
sentry cleanupCronJob forSpikeProjectionsandSpikemodels runs a tight DELETE loop viaBulkDeleteQuery._continuous_query(), creating a burst of dead tuples ondb-usage-1. This triggers a massive autovacuum that causes WAL replication delay, forcingdb-usage-replto fall back to GSCP recovery.Since
valid_datevalues are always at midnight UTC, simply running the CronJob more frequently doesn't help — all eligible rows become deletable at the same instant. We need a way to partition the rows across multiple runs.Solution
Add
id % Npartitioning toBulkDeleteQuery:Each run adds
WHERE id % 4 = {bucket}to the DELETE query, handling ~25% of eligible rows. The--partition-keyflag allows using a different column (defaults toid).Using
id(auto-increment) ensures uniform distribution, following the same principle as PR #18736 which switched spike projection batching fromorganization_id(snowflake, uneven) tosubscription.id(auto-increment, uniform).Changes
src/sentry/db/deletion.pypartition: tuple[int, int, str] | Noneparameter toBulkDeleteQuery.__init__()execute()iterator()viaFunc(F(key), Value(total), function="MOD")src/sentry/runner/commands/cleanup.py--partition-bucketCLI flag (integer, 0-based bucket index)--partition-totalCLI flag (integer, total number of buckets)--partition-keyCLI flag (default:id)cleanup()→_cleanup()→run_bulk_query_deletes()→BulkDeleteQuery()Test plan
test_partition_restriction— verifies only rows in the matching bucket are deletedtest_partition_with_datetime_restriction— combines partition + date filtertest_partition_all_buckets_cover_all_rows— verifies complete coverage across all bucketstest_iteration_with_partition— verifiesiterator()respects partition filtertest_partition_bucket_exceeds_total— validation error for bucket >= totaltest_partition_negative_bucket— validation error for negative buckettest_partition_zero_total— validation error for zero totaltest_partition_bucket_without_total— validation error when only bucket is settest_partition_total_without_bucket— validation error when only total is setRelated
accounts_spike_projectionscauses replication delay ondb-usage-1