KAN-278 chore: squash migration graph to a fresh baseline - #434
Merged
Conversation
Deletes all 469 first-party migration files (accounting, accounts, client, crm, job, operations, process, purchasing, quoting, timesheet, workflow) ahead of regenerating a fresh baseline. The historic graph carried latent cross-app ordering gaps (workflow/0157 vs job/0015, accounts/0017 vs job/0098+) that surface whenever a new dependency edge reshuffles Django's DFS leaf order. Also deletes tests that import the deleted migration modules to replay one-shot historical data repairs (effects long baked into every live DB): - apps/client/tests/test_migration_0017_backfill.py - apps/job/tests/test_onsite_rerate.py (replayed job/0102) - apps/job/tests/test_labour_reclassification.py (replayed job/0100-0101) - CostLineSubtypeBackfillRuleTests in test_labour_subtypes.py (job/0095) - XeroRawJsonMigrationHelperTests in test_xero_document_raw_json.py (accounting/0008) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
makemigrations output against current models: one 0001_baseline per app, plus 0002_baseline where cross-app FK cycles force a split (accounting, accounts). timesheet has no models and gets no migrations. Named *_baseline (not 0001_initial) because a replacing migration whose key collides with a name in any replaces list trips Django's "Cyclical squash replacement" loader check. Each migration carries a replaces list naming every old file-backed migration (464 total), so Django's check_replacements() records the baselines as applied on any up-to-date database - zero DDL on live instances, no --fake, no ledger surgery. Ghost ledger names from historic renumbering are deliberately excluded (a replaces entry a DB lacks would abort as partially applied). One hand edit: TrigramExtension() opens client/0001_baseline - the client_name_trgm_idx GinIndex needs pg_trgm at creation time (historically installed by job/0083); everything else regenerates from model Meta. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fresh databases (pytest conftest, CI, instance.sh create) get seed data from migrations, not fixtures. Five typed, idempotent, reversible seeds carry over what the deleted historic RunPython migrations provided: - accounts/0003: system automation Staff user (Staff.get_automation_user() raises without it) - job/0002: labour subtype catalogue, final state of old 0093+0098+0099; Onsite at 165.00, others at the historical fresh-install default 105.00 (CompanyDefaults.charge_out_rate no longer exists; per-instance tuning happens in the labour-rates admin UI) - workflow/0002: the 7 canonical-UUID XeroPayItems (prod backups exclude this table; Job/CostLine FK references must resolve) - workflow/0003: 9 Celery Beat schedules (old 0220+0226+0229) - crm/0002: 2 phone-call schedules (old crm 0001 seed net of 0005) The remaining ~85 historic RunPython/RunSQL migrations were backfills or env-gated one-shots - no-ops on an empty database, effects baked into every live one. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ared Schema-parity verification (old-graph fresh install vs new-graph fresh install vs the dev instance) exposed three objects on live databases that no model declares and the baseline schema does not create. Normalise live databases to model truth; IF EXISTS makes each a no-op on fresh installs. These are the only migrations in this PR that execute real DDL on live instances: - purchasing/0002: drop workflow_stock_xero_id_key, a duplicate UNIQUE on stock.xero_id (the Meta constraint unique_xero_id_stock remains) - workflow/0004: drop the orphaned varchar_pattern_ops index on companydefaults.company_name (companion of an index removed long ago) - crm/0003: drop crm_phonenumberclientmapping, an empty, code-unreferenced table from a since-renumbered historic migration After these, the dev instance's normalized schema catalogs match a fresh install exactly, except companydefaults.id being serial on existing DBs vs identity on fresh ones (Django 6 default; functionally identical, and django-solo always writes pk=1 explicitly). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…g tests Both files create Jobs; Job.save -> generate_job_number -> CompanyDefaults.get_solo(), and the singleton cannot be lazily created (shop_client is NOT NULL). They previously passed only because an earlier test in collection order had loaded the fixture; deleting the migration-replay tests changed the order and surfaced the latent dependency. Load the fixture explicitly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Baseline shrinks 4823 -> 4627: entries for deleted migration files under apps/*/migrations/ (only workflow/migrations/ was mypy-excluded). The new baseline and seed migrations are fully typed and add no entries. Docs: updating.md and restore-prod-to-nonprod.md note that dumps taken before the squash can only be migrated by a pre-squash checkout. restore-workaround-jobevent-staff-null.md met its own stated deletion condition (post-0079 prod backups everywhere) and referenced migrations that no longer exist; deleted per its own instruction. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Important Review skippedToo many files! This PR contains 493 files, which is 343 over the limit of 150. To get a review, narrow the scope: Upgrade to a paid plan to raise the limit. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (493)
You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Squashes the entire first-party migration graph (469 files, 11 apps) to a fresh baseline. Step 1 of the KAN-278 redo — a prerequisite for the Client→Company rename (separate PR).
Why
The historic graph carried latent cross-app ordering gaps (e.g.
workflow/0157deleting state models without depending onjob/0015;accounts/0017order-fragile againstjob/0098+). The graph only worked because Django's DFS leaf order happened to dodge them — any new cross-app dependency edge reshuffles that order and trips the gaps (this burned about a day of the abandoned first rename attempt). A fresh baseline erases the whole failure class.Mechanism
Fresh
0001_baseline(+0002_baselinewhere cross-app FK cycles force a split) generated bymakemigrations, each carrying areplaceslist naming every old file-backed migration. Django'scheck_replacements()records the baselines as applied on any DB whose ledger contains all replaced names — zero DDL on live databases, no--fake, no ledger surgery. Old files are deleted in this PR (all instances are fully applied, so nothing can be partially replaced). New migration names deliberately avoid every historic name (a replaces list naming the migration's own key makes Django's loader report "Cyclical squash replacement").Data that fresh installs previously got from historic
RunPythonmigrations is re-seeded by five typed, idempotent, reversible seed migrations:accounts/0003system automation user (Staff.get_automation_user()requires it)job/0002labour subtype catalogue (final state of old 0093+0098+0099)workflow/0002the 7 canonical-UUID XeroPayItems (prod backups exclude this table; FKs must resolve)workflow/00039 Celery Beat schedules (old 0220+0226+0229)crm/00022 phone-call schedules (old crm seeds, net of 0005)The remaining ~85 RunPython/RunSQL migrations were backfills/repairs of then-existing data or env-gated one-shots — verified no-ops on an empty DB; their effects are baked into every live DB.
TrigramExtension()is hand-added at the top ofclient/0001_baseline(theclient_name_trgm_idxGinIndex needs pg_trgm; historically installed byjob/0083) — the only schema statemakemigrationscould not regenerate.Every instance (prod + UAT) must be deployed to current
main(≥ 6ebc1b6) before this merges, so its ledger contains every replaced migration. An instance that misses this fails its next migrate loudly and pre-DDL with "Django tried to replace migration … but wasn't able to because some of the replaced migrations are already applied". Recovery: deploy pre-squash main, migrate, then redeploy.Schema-truth cleanups (the only observable DB changes)
Verification exposed three objects live DBs carry that the models don't declare. Per the fix-the-data posture, three post-baseline migrations (not in any
replaces, so they run everywhere;IF EXISTSno-ops on fresh DBs) normalise live DBs to model truth:purchasing/0002dropsworkflow_stock_xero_id_key— a duplicate UNIQUE onstock.xero_id(the Meta constraintunique_xero_id_stockremains).workflow/0004dropsworkflow_companydefaults_company_name_c3a41743_like— an orphaned pattern-ops index with no base index and no model backing.crm/0003dropscrm_phonenumberclientmapping— an orphaned empty table from a since-renumbered historic migration (0 rows, zero code references).Verification
companydefaults.idis aserialon existing DBs andGENERATED BY DEFAULT AS IDENTITYon fresh ones (Django 6 default; functionally identical, and django-solo always writes pk=1 explicitly).migrate --planlisted only the three cleanups; schema dump before/after differs by exactly the three dropped objects; ledger gained exactly the 20 new migration names;migrate --checkandmakemigrations --check --dry-runclean. After the cleanups, the dev instance's normalized catalogs match a fresh install exactly except the serial/identity line.bash scripts/check_mypy.shclean; baseline shrank 4823 → 4627 (orphaned migration entries; seeds added none).Deleted tests
They imported deleted migration modules to replay one-shot historical data repairs (effects long baked into every live DB):
test_migration_0017_backfill.py,test_onsite_rerate.py,test_labour_reclassification.py, plus classesCostLineSubtypeBackfillRuleTestsandXeroRawJsonMigrationHelperTests.Notes
replaces).docs/updating.md/docs/restore-prod-to-nonprod.mdupdated; the self-expiringrestore-workaround-jobevent-staff-null.mdmet its own deletion condition and is removed). E2E is unaffected — global-setup takes a fresh dump per run.replaceslists once every instance has migrated once post-merge.🤖 Generated with Claude Code