Make content_ids migration incremental and restartable - #8021
Conversation
|
|
||
| class Migration(migrations.Migration): | ||
| # Per-repository commits inside RunPython; AlterField runs afterwards. | ||
| atomic = False |
There was a problem hiding this comment.
This opens up a window, right?
But as long as nothing runs at the same time, or (in the case of ZDU) the currently running code is sufficiantly new there is nothing to introduce new repository versions with NULL. I think this is actually fine.
There was a problem hiding this comment.
Maybe, if the user is doing a big jump and trying to do ZDU they might run into a problem if an old worker gets a task to create a new repo-vesion. Theoretically if that were to happen and then the ALTER TABLE call fails, they could just rerun the migration again and we would only need to update that one new repo-version that slipped through.
There was a problem hiding this comment.
"big jump and ZDU" is not supported. Let's please not use vague language around that.
There was a problem hiding this comment.
Gotcha, then there should be no window if they upgrade properly.
|
|
||
| import django.contrib.postgres.fields | ||
| from django.db import migrations, models | ||
| from django.db import migrations, models, transaction |
There was a problem hiding this comment.
I was thinking about something along the lines of
UPDATE core.repositoryversion AS rv SET content_ids=(SELECT ARRAY_AGG(content_id) FROM core.repositoryversion_content WHERE version_added <= rv.version and version_removed > rv.version) WHERE rv.content_ids is NULL;
Maybe needs a few more joins and some care for version_removed = NULL.
Am oversimplifying the challenge here?
Maybe one can even batch that by adding a LIMIT 1000 clause.
There was a problem hiding this comment.
Updated the query to be the exact SQL version of _content_relationships. Calculating the memberships can take a little while so not sure if batching them would be good if we want to avoid timeouts.
Perform the cache populating entirely in SQL and add per repository commits. Assisted By: Cursor Grok 4.6 Co-authored-by: Cursor <cursoragent@cursor.com>
87e316f to
ceba57e
Compare
Backport to 3.116: 💚 backport PR created✅ Backport PR branch: Backported as #8031 🤖 @patchback |
2 participants
Summary
content_idsdata fill to compute each repository version in Postgres from the previous version (plus added, minus removed) instead of an N+1 ORM loop that ships UUID arrays through Python.atomic = False) so a DBaaS statement/session timeout does not roll back already-filled repos. Re-running migrate continues from remaining NULLs, thenAlterFieldmakes the column required.bulk_updatebatches does not fix the per-version recompute or the single wrapping transaction.Test plan
Made with Cursor