Skip to content

Commit db22cfe

Browse files
wedamijawedamija
authored
Update docs on how to delete tables and columns (#11897)
This uses the new operations from getsentry/sentry#81063 getsentry/sentry#81098
1 parent e54cfc0 commit db22cfe

File tree

1 file changed

+215
-83
lines changed
  • develop-docs/api-server/application-domains/database-migrations

1 file changed

+215
-83
lines changed

‎develop-docs/api-server/application-domains/database-migrations/index.mdx‎

Lines changed: 215 additions & 83 deletions
Original file line numberDiff line numberDiff line change
@@ -145,144 +145,276 @@ This is complicated due to our deploy process. When we deploy, we run migrations
145145

146146
To avoid this, follow these steps:
147147

148-
- Mark the column as nullable if it isn't, and create a migration. (ex. `BoundedIntegerField(null=True)`)
149-
- Deploy.
150-
- Remove the column from the model, but in the migration make sure we only mark the state as removed.
151-
- Deploy.
152-
- Finally, create a migration that deletes the column.
148+
- Make a PR to remove all uses of the column in the codebase in a separate PR. This mostly helps with code cleanliness. This should be merged ahead of the migration prs, but we don't need to worry about whether it is deployed first.
149+
- Make another PR that:
150+
- Checks if the column is either not nullable, or doesn't have a `db_default` set. If either of these is true, then make it nullable via `null=True`.
151+
- If the column is a foreign key, remove the database level foreign key constraint it by setting `db_constraint=False`.
152+
- Remove the column and in the generated migration use `SafeRemoveField(..., deletion_action=DeletionAction.MOVE_TO_PENDING)` to replace `RemoveField(...)`. This only marks the state for the column as removed.
153+
- Combine these migrations together to save making multiple deploys
154+
- Deploy your migration changes. It's important that all previous pull requests are in production before we remove the actual column from the table.
155+
- Make a pull request that create a new migration that has the same `SafeRemoveField` operation as before, but set `deletion_action=DeletionAction.DELETE` instead. This deletes the actual column from the table in Postgres.
156+
- Deploy the drop column migration.
157+
158+
Here's an example of removing the `project` column from this model. It is both a foreign key and not null:
153159

154-
Here's an example of removing columns that were already nullable. First we remove the columns from the model, and then modify the migration to only update the state and make no database operations.
160+
```python
161+
@region_silo_model
162+
class TestModel(Model):
163+
__relocation_scope__ = RelocationScope.Excluded
164+
165+
project = FlexibleForeignKey("sentry.Project")
166+
name = models.TextField()
167+
168+
class Meta:
169+
app_label = "uptime"
170+
db_table = "uptime_testmodel"
171+
```
172+
173+
First, we remove all references to this field from the codebase. This is best done as a separate pr to keep things clean.
174+
175+
Next we produce two migrations, in individual prs that we will deploy separately.
176+
177+
First PR
178+
```python
179+
# <Field removed from model and code completely>
180+
181+
# First migration
182+
# ... Migration boilerplate ...
183+
operations = [
184+
migrations.AlterField(
185+
model_name="testmodel",
186+
name="project",
187+
field=sentry.db.models.fields.foreignkey.FlexibleForeignKey(
188+
db_constraint=False,
189+
null=True,
190+
on_delete=django.db.models.deletion.CASCADE,
191+
to="sentry.project",
192+
),
193+
),
194+
SafeRemoveField(model_name="testmodel", name="project", deletion_action=DeletionAction.MOVE_TO_PENDING),
195+
]
196+
# ... Migration boilerplate ...
197+
```
155198

199+
Second PR:
156200
```python
201+
# Second migration
202+
# ... Migration boilerplate ...
157203
operations = [
158-
migrations.SeparateDatabaseAndState(
159-
database_operations=[],
160-
state_operations=[
161-
migrations.RemoveField(model_name="alertrule", name="alert_threshold"),
162-
migrations.RemoveField(model_name="alertrule", name="resolve_threshold"),
163-
migrations.RemoveField(model_name="alertrule", name="threshold_type"),
164-
],
165-
)
204+
SafeRemoveField(model_name="testmodel", name="project", deletion_action=DeletionAction.DELETE),
166205
]
206+
# ... Migration boilerplate ...
167207
```
168208

169-
Once this is deployed, we can then deploy the actual column deletion. This pr will have only a migration, since Django no longer knows about these fields. Note that the reverse SQL is only for dev, so it's fine to not assign a default or do any sort of backfill:
209+
So once we have these two prs, we merge/deploy the first, and then the second and then the table is fully removed.
210+
211+
So to recap the steps here:
212+
- Remove all references to the column in the code in a separate pull request and merge. Doesn't matter if this deploys before the next step or not.
213+
- If the column has a foreign key constraint them remove it. If it's not null and has no `db_default` then mark it as nullable. Then delete the column using `SafeRemoveField(..., deletion_action=DeletionAction.MOVE_TO_PENDING)`. These operations can be in the same migration to save time.
214+
- Deploy all previous before continuing.
215+
- Remove the column from the table in from Postgres using `SafeRemoveField(..., deletion_action=DeletionAction.DELETE),`
216+
217+
If you're comfortable producing these prs and deploying them, then stop here. Otherwise, this next section covers how to produce them in more detail.
218+
219+
To produce the first migration, we need to remove the db level foreign key constraint, make the column nullable and remove the column from the codebase. To remove the db level foreign key constraints and mark the column nullable we add `db_constraint=False, null=True` to this column and generate a migration:
170220

171221
```python
222+
# Model change
223+
...
224+
project = FlexibleForeignKey("sentry.Project", db_constraint=False, null=True)
225+
...
226+
227+
# Migration operations
172228
operations = [
173-
migrations.SeparateDatabaseAndState(
174-
database_operations=[
175-
migrations.RunSQL(
176-
"""
177-
ALTER TABLE "sentry_alertrule" DROP COLUMN "alert_threshold";
178-
ALTER TABLE "sentry_alertrule" DROP COLUMN "resolve_threshold";
179-
ALTER TABLE "sentry_alertrule" DROP COLUMN "threshold_type";
180-
""",
181-
reverse_sql="""
182-
ALTER TABLE "sentry_alertrule" ADD COLUMN "alert_threshold" smallint NULL;
183-
ALTER TABLE "sentry_alertrule" ADD COLUMN "resolve_threshold" int NULL;
184-
ALTER TABLE "sentry_alertrule" ADD COLUMN "threshold_type" int NULL;
185-
""",
186-
hints={"tables": ["sentry_alertrule"]},
187-
)
188-
],
189-
state_operations=[],
190-
)
229+
migrations.AlterField(
230+
model_name="testmodel",
231+
name="project",
232+
field=sentry.db.models.fields.foreignkey.FlexibleForeignKey(
233+
db_constraint=False,
234+
null=True,
235+
on_delete=django.db.models.deletion.CASCADE,
236+
to="sentry.project",
237+
),
238+
),
239+
]
240+
```
241+
242+
Once we've done this, we can now remove the column from the model and generate the migration to remove it. The generated migration looks like this:
243+
244+
```python
245+
operations = [
246+
migrations.RemoveField(model_name="testmodel", name="project"),
247+
]
248+
```
249+
250+
Django doesn't know about the `SafeRemoveField` operation, so we replace it with that instead. This allows us to remove all state related to the column, but defer deleting it until a later migration. So this becomes
251+
252+
```python
253+
operations = [
254+
SafeRemoveField(model_name="testmodel", name="project", deletion_action=DeletionAction.MOVE_TO_PENDING),
255+
]
256+
```
257+
258+
So now as a final step, we can combine these operations into a single migration, which is the first migration we want to deploy.
259+
260+
```python
261+
operations = [
262+
migrations.AlterField(
263+
model_name="testmodel",
264+
name="project",
265+
field=sentry.db.models.fields.foreignkey.FlexibleForeignKey(
266+
db_constraint=False,
267+
null=True,
268+
on_delete=django.db.models.deletion.CASCADE,
269+
to="sentry.project",
270+
),
271+
),
272+
SafeRemoveField(model_name="testmodel", name="project", deletion_action=DeletionAction.MOVE_TO_PENDING),
273+
]
274+
```
275+
276+
To produce the second migration we generate an empty migration (`sentry django makemigrations <your_app> --empty`), then use the same `SafeRemoveField` command from the previous migration, but change the deletion_action to `DeletionAction.DELETE`. This operation will remove the column from the table in Postgres:
277+
278+
```python
279+
operations = [
280+
SafeRemoveField(model_name="testmodel", name="project", deletion_action=DeletionAction.DELETE),
191281
]
192282
```
193283

194284
### Deleting Tables
195285

196286
Extra care is needed here if the table is referenced as a foreign key in other tables. In that case, first remove the foreign key columns in the other tables, then come back to this step.
197287

198-
- Remove any database level foreign key constraints from this table to other tables by setting `db_constraint=False` on the columns.
199-
- Deploy
200-
- Remove the model and all references from the sentry codebase. Make sure that the migration only marks the state as removed.
201-
- Deploy.
202-
- Create a migrations that deletes the table.
288+
- Make a pull request to remove all uses of the model in the codebase in a separate pull request. This mostly helps with code cleanliness. This should be merged ahead of the migration pull requests, but we don't need to worry about whether it is deployed first.
289+
- Make another pull request to:
290+
- Remove any database level foreign key constraints from this table to other tables by setting `db_constraint=False` on the columns.
291+
- Remove the model and in the generated migration use `SafeDeleteModel(..., deletion_action=DeletionAction.MOVE_TO_PENDING)` to replace `DeleteModel(...)`. This only marks the state for the model as removed.
292+
- Deploy. It's important that all previous pull requests are in production before we remove the actual table.
293+
- Make a pull request that creates a new migration that has the same `SafeDeleteModel` operation as before, but set `deletion_action=DeletionAction.DELETE` instead. This deletes the actual table from Postgres.
203294
- Deploy
204295

205296
Here's an example of removing this model:
206297

207298
```python
208-
class AlertRuleTriggerAction(Model):
209-
alert_rule_trigger = FlexibleForeignKey("sentry.AlertRuleTrigger")
210-
integration = FlexibleForeignKey("sentry.Integration", null=True)
211-
type = models.SmallIntegerField()
212-
target_type = models.SmallIntegerField()
213-
# Identifier used to perform the action on a given target
214-
target_identifier = models.TextField(null=True)
215-
# Human readable name to display in the UI
216-
target_display = models.TextField(null=True)
217-
date_added = models.DateTimeField(default=timezone.now)
299+
@region_silo_model
300+
class TestModel(Model):
301+
__relocation_scope__ = RelocationScope.Excluded
302+
303+
project = FlexibleForeignKey("sentry.Project")
304+
name = models.TextField()
218305

219306
class Meta:
220-
app_label = "sentry"
221-
db_table = "sentry_alertruletriggeraction"
307+
app_label = "uptime"
308+
db_table = "uptime_testmodel"
222309
```
223310

224-
First we checked that it's not referenced by any other models, and it's not. Next we need to remove and db level foreign key constraints. To do this, we change these two columns and generate a migration:
311+
First, we remove all references to this model from the codebase, including making sure that it's not referenced by any other models. This is best done as a separate pr to keep things clean.
312+
313+
Next we produce two migrations, in individual prs that we will deploy separately.
314+
315+
First PR
316+
```python
317+
# <Model removed from code completely>
318+
319+
# First migration
320+
# ... Migration boilerplate ...
321+
operations = [
322+
migrations.AlterField(
323+
model_name="testmodel",
324+
name="project",
325+
field=sentry.db.models.fields.foreignkey.FlexibleForeignKey(
326+
db_constraint=False,
327+
on_delete=django.db.models.deletion.CASCADE,
328+
to="sentry.project",
329+
),
330+
),
331+
SafeDeleteModel(name="TestModel", deletion_action=DeletionAction.MOVE_TO_PENDING),
332+
]
333+
# ... Migration boilerplate ...
334+
```
225335

336+
Second PR:
226337
```python
227-
alert_rule_trigger = FlexibleForeignKey("sentry.AlertRuleTrigger", db_constraint=False)
228-
integration = FlexibleForeignKey("sentry.Integration", null=True, db_constraint=False)
338+
# Second migration
339+
# ... Migration boilerplate ...
340+
operations = [
341+
SafeDeleteModel(name="TestModel", deletion_action=DeletionAction.DELETE),
342+
]
343+
# ... Migration boilerplate ...
229344
```
230345

231-
The operations in the migration look like
346+
So once we have these two prs, we merge/deploy the first, and then the second and then the table is fully removed.
347+
348+
So to recap the steps here:
349+
- Remove all references to the model in the code in a separate pull request and merge. Doesn't matter if this deploys before the next step or not.
350+
- Remove any foreign key constraints and delete the model using `SafeDeleteModel(..., deletion_action=DeletionAction.MOVE_TO_PENDING)`. These operations can be in the same migration to save time.
351+
- Deploy all previous before continuing.
352+
- Remove the table from Postgres using `SafeDeleteModel(..., deletion_action=DeletionAction.DELETE),`
353+
354+
If you're comfortable producing these prs and deploying them, then stop here. Otherwise, this next section covers how to produce them in more detail.
355+
356+
To produce the first migration, we need to remove any db level foreign key constraints and remove the table from the codebase. To remove the db level foreign key constraints we add `db_constraint=False` to this column and generate a migration:
232357

358+
```python
359+
project = FlexibleForeignKey("sentry.Project", db_constraint=False)
360+
```
361+
362+
This produces a migration with operations like:
233363
```python
234364
operations = [
235365
migrations.AlterField(
236-
model_name='alertruletriggeraction',
237-
name='alert_rule_trigger',
238-
field=sentry.db.models.fields.foreignkey.FlexibleForeignKey(db_constraint=False, on_delete=django.db.models.deletion.CASCADE, to='sentry.AlertRuleTrigger'),
239-
),
240-
migrations.AlterField(
241-
model_name='alertruletriggeraction',
242-
name='integration',
243-
field=sentry.db.models.fields.foreignkey.FlexibleForeignKey(db_constraint=False, null=True, on_delete=django.db.models.deletion.CASCADE, to='sentry.Integration'),
366+
model_name="testmodel",
367+
name="project",
368+
field=sentry.db.models.fields.foreignkey.FlexibleForeignKey(
369+
db_constraint=False,
370+
on_delete=django.db.models.deletion.CASCADE,
371+
to="sentry.project",
372+
),
244373
),
245374
]
246375
```
247376

248-
And we can see the sql it generates just drops the FK constaints
377+
Next, we remove the model from code and generate the migration to remove it. The generated migration looks like this:
249378

250379
```python
251-
BEGIN;
252-
SET CONSTRAINTS "a875987ae7debe6be88869cb2eebcdc5" IMMEDIATE; ALTER TABLE "sentry_alertruletriggeraction" DROP CONSTRAINT "a875987ae7debe6be88869cb2eebcdc5";
253-
SET CONSTRAINTS "sentry_integration_id_14286d876e86361c_fk_sentry_integration_id" IMMEDIATE; ALTER TABLE "sentry_alertruletriggeraction" DROP CONSTRAINT "sentry_integration_id_14286d876e86361c_fk_sentry_integration_id";
254-
COMMIT;
380+
operations = [
381+
migrations.DeleteModel(name="TestModel"),
382+
]
255383
```
256384

257-
So now we deploy this and move onto the next stage.
258-
259-
The next stage involves removing all references to the model from the codebase. So we do that, and then we generate a migration that removes the model from the migration state, but not the database. The operations in this migration look like
385+
Django doesn't know about the `SafeDeleteModel` operation, so we replace it with that instead. This allows us to remove all state related to the model, but defer deleting it until a later migration. So this becomes
260386

261387
```python
262388
operations = [
263-
migrations.SeparateDatabaseAndState(
264-
state_operations=[migrations.DeleteModel(name="AlertRuleTriggerAction")],
265-
database_operations=[],
266-
)
389+
SafeDeleteModel(name="TestModel", deletion_action=DeletionAction.MOVE_TO_PENDING),
267390
]
268391
```
269392

270-
and the generated SQL shows no database changes occurring. So now we deploy this and move into the final step.
393+
So now as a final step, we can combine these operations into a single migration, which is the first migration we want to deploy.
394+
```python
395+
operations = [
396+
migrations.AlterField(
397+
model_name="testmodel",
398+
name="project",
399+
field=sentry.db.models.fields.foreignkey.FlexibleForeignKey(
400+
db_constraint=False,
401+
on_delete=django.db.models.deletion.CASCADE,
402+
to="sentry.project",
403+
),
404+
),
405+
SafeDeleteModel(name="TestModel", deletion_action=DeletionAction.MOVE_TO_PENDING),
406+
]
407+
```
271408

272-
In this last step, we just want to manually write DDL to remove the table. So we use `sentry django makemigrations --empty` to produce an empty migration, and then modify the operations to be like:
409+
To produce the second migration we generate an empty migration (`sentry django makemigrations <your_app> --empty`), then use the same `SafeDeleteModel` command from the previous migration, but change the deletion_action to `DeletionAction.DELETE`. This operation will remove the table from Postgres:
273410

274411
```python
275412
operations = [
276-
migrations.RunSQL(
277-
"""
278-
DROP TABLE "sentry_alertruletriggeraction";
279-
""",
280-
reverse_sql="CREATE TABLE sentry_alertruletriggeraction (fake_col int)", # We just create a fake table here so that the DROP will work if we roll back the migration.
281-
)
413+
SafeDeleteModel(name="TestModel", deletion_action=DeletionAction.DELETE),
282414
]
283415
```
284416

285-
Then we deploy this and we're done.
417+
This second PR will contain only the migration and related boilerplate.
286418

287419
### Foreign Keys
288420

0 commit comments

Comments
 (0)