/
migrations.txt
981 lines (733 loc) · 40.1 KB
/
migrations.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
==========
Migrations
==========
.. module:: django.db.migrations
:synopsis: Schema migration support for Django models
Migrations are Django's way of propagating changes you make to your models
(adding a field, deleting a model, etc.) into your database schema. They're
designed to be mostly automatic, but you'll need to know when to make
migrations, when to run them, and the common problems you might run into.
The Commands
============
There are several commands which you will use to interact with migrations
and Django's handling of database schema:
* :djadmin:`migrate`, which is responsible for applying and unapplying
migrations.
* :djadmin:`makemigrations`, which is responsible for creating new migrations
based on the changes you have made to your models.
* :djadmin:`sqlmigrate`, which displays the SQL statements for a migration.
* :djadmin:`showmigrations`, which lists a project's migrations and their
status.
You should think of migrations as a version control system for your database
schema. ``makemigrations`` is responsible for packaging up your model changes
into individual migration files - analogous to commits - and ``migrate`` is
responsible for applying those to your database.
The migration files for each app live in a "migrations" directory inside
of that app, and are designed to be committed to, and distributed as part
of, its codebase. You should be making them once on your development machine
and then running the same migrations on your colleagues' machines, your
staging machines, and eventually your production machines.
.. note::
It is possible to override the name of the package which contains the
migrations on a per-app basis by modifying the :setting:`MIGRATION_MODULES`
setting.
Migrations will run the same way on the same dataset and produce consistent
results, meaning that what you see in development and staging is, under the
same circumstances, exactly what will happen in production.
Django will make migrations for any change to your models or fields - even
options that don't affect the database - as the only way it can reconstruct
a field correctly is to have all the changes in the history, and you might
need those options in some data migrations later on (for example, if you've
set custom validators).
Backend Support
===============
Migrations are supported on all backends that Django ships with, as well
as any third-party backends if they have programmed in support for schema
alteration (done via the :doc:`SchemaEditor </ref/schema-editor>` class).
However, some databases are more capable than others when it comes to
schema migrations; some of the caveats are covered below.
PostgreSQL
----------
PostgreSQL is the most capable of all the databases here in terms of schema
support.
The only caveat is that prior to PostgreSQL 11, adding columns with default
values causes a full rewrite of the table, for a time proportional to its size.
For this reason, it's recommended you always create new columns with
``null=True``, as this way they will be added immediately.
MySQL
-----
MySQL lacks support for transactions around schema alteration operations,
meaning that if a migration fails to apply you will have to manually unpick
the changes in order to try again (it's impossible to roll back to an
earlier point).
In addition, MySQL will fully rewrite tables for almost every schema operation
and generally takes a time proportional to the number of rows in the table to
add or remove columns. On slower hardware this can be worse than a minute per
million rows - adding a few columns to a table with just a few million rows
could lock your site up for over ten minutes.
Finally, MySQL has relatively small limits on name lengths for columns, tables
and indexes, as well as a limit on the combined size of all columns an index
covers. This means that indexes that are possible on other backends will
fail to be created under MySQL.
SQLite
------
SQLite has very little built-in schema alteration support, and so Django
attempts to emulate it by:
* Creating a new table with the new schema
* Copying the data across
* Dropping the old table
* Renaming the new table to match the original name
This process generally works well, but it can be slow and occasionally
buggy. It is not recommended that you run and migrate SQLite in a
production environment unless you are very aware of the risks and
its limitations; the support Django ships with is designed to allow
developers to use SQLite on their local machines to develop less complex
Django projects without the need for a full database.
Workflow
========
Django can create migrations for you. Make changes to your models - say, add a
field and remove a model - and then run :djadmin:`makemigrations`::
$ python manage.py makemigrations
Migrations for 'books':
books/migrations/0003_auto.py:
- Alter field author on book
Your models will be scanned and compared to the versions currently
contained in your migration files, and then a new set of migrations
will be written out. Make sure to read the output to see what
``makemigrations`` thinks you have changed - it's not perfect, and for
complex changes it might not be detecting what you expect.
Once you have your new migration files, you should apply them to your
database to make sure they work as expected::
$ python manage.py migrate
Operations to perform:
Apply all migrations: books
Running migrations:
Rendering model states... DONE
Applying books.0003_auto... OK
Once the migration is applied, commit the migration and the models change
to your version control system as a single commit - that way, when other
developers (or your production servers) check out the code, they'll
get both the changes to your models and the accompanying migration at the
same time.
If you want to give the migration(s) a meaningful name instead of a generated
one, you can use the :option:`makemigrations --name` option::
$ python manage.py makemigrations --name changed_my_model your_app_label
Version control
---------------
Because migrations are stored in version control, you'll occasionally
come across situations where you and another developer have both committed
a migration to the same app at the same time, resulting in two migrations
with the same number.
Don't worry - the numbers are just there for developers' reference, Django
just cares that each migration has a different name. Migrations specify which
other migrations they depend on - including earlier migrations in the same
app - in the file, so it's possible to detect when there's two new migrations
for the same app that aren't ordered.
When this happens, Django will prompt you and give you some options. If it
thinks it's safe enough, it will offer to automatically linearize the two
migrations for you. If not, you'll have to go in and modify the migrations
yourself - don't worry, this isn't difficult, and is explained more in
:ref:`migration-files` below.
Transactions
============
On databases that support DDL transactions (SQLite and PostgreSQL), all
migration operations will run inside a single transaction by default. In
contrast, if a database doesn't support DDL transactions (e.g. MySQL, Oracle)
then all operations will run without a transaction.
You can prevent a migration from running in a transaction by setting the
``atomic`` attribute to ``False``. For example::
from django.db import migrations
class Migration(migrations.Migration):
atomic = False
It's also possible to execute parts of the migration inside a transaction using
:func:`~django.db.transaction.atomic()` or by passing ``atomic=True`` to
:class:`~django.db.migrations.operations.RunPython`. See
:ref:`non-atomic-migrations` for more details.
Dependencies
============
While migrations are per-app, the tables and relationships implied by
your models are too complex to be created for one app at a time. When you make
a migration that requires something else to run - for example, you add a
``ForeignKey`` in your ``books`` app to your ``authors`` app - the resulting
migration will contain a dependency on a migration in ``authors``.
This means that when you run the migrations, the ``authors`` migration runs
first and creates the table the ``ForeignKey`` references, and then the migration
that makes the ``ForeignKey`` column runs afterwards and creates the constraint.
If this didn't happen, the migration would try to create the ``ForeignKey``
column without the table it's referencing existing and your database would
throw an error.
This dependency behavior affects most migration operations where you
restrict to a single app. Restricting to a single app (either in
``makemigrations`` or ``migrate``) is a best-efforts promise, and not
a guarantee; any other apps that need to be used to get dependencies correct
will be.
Apps without migrations must not have relations (``ForeignKey``,
``ManyToManyField``, etc.) to apps with migrations. Sometimes it may work, but
it's not supported.
.. _migration-files:
Migration files
===============
Migrations are stored as an on-disk format, referred to here as
"migration files". These files are actually normal Python files with an
agreed-upon object layout, written in a declarative style.
A basic migration file looks like this::
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [('migrations', '0001_initial')]
operations = [
migrations.DeleteModel('Tribble'),
migrations.AddField('Author', 'rating', models.IntegerField(default=0)),
]
What Django looks for when it loads a migration file (as a Python module) is
a subclass of ``django.db.migrations.Migration`` called ``Migration``. It then
inspects this object for four attributes, only two of which are used
most of the time:
* ``dependencies``, a list of migrations this one depends on.
* ``operations``, a list of ``Operation`` classes that define what this
migration does.
The operations are the key; they are a set of declarative instructions which
tell Django what schema changes need to be made. Django scans them and
builds an in-memory representation of all of the schema changes to all apps,
and uses this to generate the SQL which makes the schema changes.
That in-memory structure is also used to work out what the differences are
between your models and the current state of your migrations; Django runs
through all the changes, in order, on an in-memory set of models to come
up with the state of your models last time you ran ``makemigrations``. It
then uses these models to compare against the ones in your ``models.py`` files
to work out what you have changed.
You should rarely, if ever, need to edit migration files by hand, but
it's entirely possible to write them manually if you need to. Some of the
more complex operations are not autodetectable and are only available via
a hand-written migration, so don't be scared about editing them if you have to.
Custom fields
-------------
You can't modify the number of positional arguments in an already migrated
custom field without raising a ``TypeError``. The old migration will call the
modified ``__init__`` method with the old signature. So if you need a new
argument, please create a keyword argument and add something like
``assert 'argument_name' in kwargs`` in the constructor.
.. _using-managers-in-migrations:
Model managers
--------------
You can optionally serialize managers into migrations and have them available
in :class:`~django.db.migrations.operations.RunPython` operations. This is done
by defining a ``use_in_migrations`` attribute on the manager class::
class MyManager(models.Manager):
use_in_migrations = True
class MyModel(models.Model):
objects = MyManager()
If you are using the :meth:`~django.db.models.from_queryset` function to
dynamically generate a manager class, you need to inherit from the generated
class to make it importable::
class MyManager(MyBaseManager.from_queryset(CustomQuerySet)):
use_in_migrations = True
class MyModel(models.Model):
objects = MyManager()
Please refer to the notes about :ref:`historical-models` in migrations to see
the implications that come along.
Initial migrations
------------------
.. attribute:: Migration.initial
The "initial migrations" for an app are the migrations that create the first
version of that app's tables. Usually an app will have one initial migration,
but in some cases of complex model interdependencies it may have two or more.
Initial migrations are marked with an ``initial = True`` class attribute on the
migration class. If an ``initial`` class attribute isn't found, a migration
will be considered "initial" if it is the first migration in the app (i.e. if
it has no dependencies on any other migration in the same app).
When the :option:`migrate --fake-initial` option is used, these initial
migrations are treated specially. For an initial migration that creates one or
more tables (``CreateModel`` operation), Django checks that all of those tables
already exist in the database and fake-applies the migration if so. Similarly,
for an initial migration that adds one or more fields (``AddField`` operation),
Django checks that all of the respective columns already exist in the database
and fake-applies the migration if so. Without ``--fake-initial``, initial
migrations are treated no differently from any other migration.
.. _migration-history-consistency:
History consistency
-------------------
As previously discussed, you may need to linearize migrations manually when two
development branches are joined. While editing migration dependencies, you can
inadvertently create an inconsistent history state where a migration has been
applied but some of its dependencies haven't. This is a strong indication that
the dependencies are incorrect, so Django will refuse to run migrations or make
new migrations until it's fixed. When using multiple databases, you can use the
:meth:`allow_migrate` method of :ref:`database routers
<topics-db-multi-db-routing>` to control which databases
:djadmin:`makemigrations` checks for consistent history.
Adding migrations to apps
=========================
New apps come preconfigured to accept migrations, and so you can add migrations
by running :djadmin:`makemigrations` once you've made some changes.
If your app already has models and database tables, and doesn't have migrations
yet (for example, you created it against a previous Django version), you'll
need to convert it to use migrations by running::
$ python manage.py makemigrations your_app_label
This will make a new initial migration for your app. Now, run ``python
manage.py migrate --fake-initial``, and Django will detect that you have an
initial migration *and* that the tables it wants to create already exist, and
will mark the migration as already applied. (Without the :option:`migrate
--fake-initial` flag, the command would error out because the tables it wants
to create already exist.)
Note that this only works given two things:
* You have not changed your models since you made their tables. For migrations
to work, you must make the initial migration *first* and then make changes,
as Django compares changes against migration files, not the database.
* You have not manually edited your database - Django won't be able to detect
that your database doesn't match your models, you'll just get errors when
migrations try to modify those tables.
.. _reversing-migrations:
Reversing migrations
====================
Migrations can be reversed with :djadmin:`migrate` by passing the number of the
previous migration. For example, to reverse migration ``books.0003``:
.. console::
$ python manage.py migrate books 0002
Operations to perform:
Target specific migration: 0002_auto, from books
Running migrations:
Rendering model states... DONE
Unapplying books.0003_auto... OK
If you want to reverse all migrations applied for an app, use the name
``zero``:
.. console::
$ python manage.py migrate books zero
Operations to perform:
Unapply all migrations: books
Running migrations:
Rendering model states... DONE
Unapplying books.0002_auto... OK
Unapplying books.0001_initial... OK
A migration is irreversible if it contains any irreversible operations.
Attempting to reverse such migrations will raise ``IrreversibleError``:
.. console::
$ python manage.py migrate books 0002
Operations to perform:
Target specific migration: 0002_auto, from books
Running migrations:
Rendering model states... DONE
Unapplying books.0003_auto...Traceback (most recent call last):
django.db.migrations.exceptions.IrreversibleError: Operation <RunSQL sql='DROP TABLE demo_books'> in books.0003_auto is not reversible
.. _historical-models:
Historical models
=================
When you run migrations, Django is working from historical versions of your
models stored in the migration files. If you write Python code using the
:class:`~django.db.migrations.operations.RunPython` operation, or if you have
``allow_migrate`` methods on your database routers, you **need to use** these
historical model versions rather than importing them directly.
.. warning::
If you import models directly rather than using the historical models,
your migrations *may work initially* but will fail in the future when you
try to re-run old migrations (commonly, when you set up a new installation
and run through all the migrations to set up the database).
This means that historical model problems may not be immediately obvious.
If you run into this kind of failure, it's OK to edit the migration to use
the historical models rather than direct imports and commit those changes.
Because it's impossible to serialize arbitrary Python code, these historical
models will not have any custom methods that you have defined. They will,
however, have the same fields, relationships, managers (limited to those with
``use_in_migrations = True``) and ``Meta`` options (also versioned, so they may
be different from your current ones).
.. warning::
This means that you will NOT have custom ``save()`` methods called on objects
when you access them in migrations, and you will NOT have any custom
constructors or instance methods. Plan appropriately!
References to functions in field options such as ``upload_to`` and
``limit_choices_to`` and model manager declarations with managers having
``use_in_migrations = True`` are serialized in migrations, so the functions and
classes will need to be kept around for as long as there is a migration
referencing them. Any :doc:`custom model fields </howto/custom-model-fields>`
will also need to be kept, since these are imported directly by migrations.
In addition, the concrete base classes of the model are stored as pointers, so
you must always keep base classes around for as long as there is a migration
that contains a reference to them. On the plus side, methods and managers from
these base classes inherit normally, so if you absolutely need access to these
you can opt to move them into a superclass.
To remove old references, you can :ref:`trim migrations <migration-trimming>`
or, if there aren't many references, copy them into the migration files.
.. _migrations-removing-model-fields:
Considerations when removing model fields
=========================================
Similar to the "references to historical functions" considerations described in
the previous section, removing custom model fields from your project or
third-party app will cause a problem if they are referenced in old migrations.
To help with this situation, Django provides some model field attributes to
assist with model field deprecation using the :doc:`system checks framework
</topics/checks>`.
Add the ``system_check_deprecated_details`` attribute to your model field
similar to the following::
class IPAddressField(Field):
system_check_deprecated_details = {
'msg': (
'IPAddressField has been deprecated. Support for it (except '
'in historical migrations) will be removed in Django 1.9.'
),
'hint': 'Use GenericIPAddressField instead.', # optional
'id': 'fields.W900', # pick a unique ID for your field.
}
After a deprecation period of your choosing (two or three feature releases for
fields in Django itself), change the ``system_check_deprecated_details``
attribute to ``system_check_removed_details`` and update the dictionary similar
to::
class IPAddressField(Field):
system_check_removed_details = {
'msg': (
'IPAddressField has been removed except for support in '
'historical migrations.'
),
'hint': 'Use GenericIPAddressField instead.',
'id': 'fields.E900', # pick a unique ID for your field.
}
You should keep the field's methods that are required for it to operate in
database migrations such as ``__init__()``, ``deconstruct()``, and
``get_internal_type()``. Keep this stub field for as long as any migrations
which reference the field exist. For example, after removing migrations by
trimming or squashing them, you should be able to remove the field completely.
.. _data-migrations:
Data Migrations
===============
As well as changing the database schema, you can also use migrations to change
the data in the database itself, in conjunction with the schema if you want.
Migrations that alter data are usually called "data migrations"; they're best
written as separate migrations, sitting alongside your schema migrations.
Django can't automatically generate data migrations for you, as it does with
schema migrations, but it's not very hard to write them. Migration files in
Django are made up of :doc:`Operations </ref/migration-operations>`, and
the main operation you use for data migrations is
:class:`~django.db.migrations.operations.RunPython`.
To start, make an empty migration file you can work from (Django will put
the file in the right place, suggest a name, and add dependencies for you)::
python manage.py makemigrations --empty yourappname
Then, open up the file; it should look something like this::
# Generated by Django A.B on YYYY-MM-DD HH:MM
from django.db import migrations
class Migration(migrations.Migration):
dependencies = [
('yourappname', '0001_initial'),
]
operations = [
]
Now, all you need to do is create a new function and have
:class:`~django.db.migrations.operations.RunPython` use it.
:class:`~django.db.migrations.operations.RunPython` expects a callable as its argument
which takes two arguments - the first is an :doc:`app registry
</ref/applications/>` that has the historical versions of all your models
loaded into it to match where in your history the migration sits, and the
second is a :doc:`SchemaEditor </ref/schema-editor>`, which you can use to
manually effect database schema changes (but beware, doing this can confuse
the migration autodetector!)
Let's write a migration that populates our new ``name`` field with the combined
values of ``first_name`` and ``last_name`` (we've come to our senses and
realized that not everyone has first and last names). All we need to do is use
the historical model and iterate over the rows::
from django.db import migrations
def combine_names(apps, schema_editor):
# We can't import the Person model directly as it may be a newer
# version than this migration expects. We use the historical version.
Person = apps.get_model('yourappname', 'Person')
for person in Person.objects.all():
person.name = '%s %s' % (person.first_name, person.last_name)
person.save()
class Migration(migrations.Migration):
dependencies = [
('yourappname', '0001_initial'),
]
operations = [
migrations.RunPython(combine_names),
]
Once that's done, we can run ``python manage.py migrate`` as normal and the
data migration will run in place alongside other migrations.
You can pass a second callable to
:class:`~django.db.migrations.operations.RunPython` to run whatever logic you
want executed when migrating backwards. If this callable is omitted, migrating
backwards will raise an exception.
Accessing models from other apps
--------------------------------
When writing a ``RunPython`` function that uses models from apps other than the
one in which the migration is located, the migration's ``dependencies``
attribute should include the latest migration of each app that is involved,
otherwise you may get an error similar to: ``LookupError: No installed app
with label 'myappname'`` when you try to retrieve the model in the ``RunPython``
function using ``apps.get_model()``.
In the following example, we have a migration in ``app1`` which needs to use
models in ``app2``. We aren't concerned with the details of ``move_m1`` other
than the fact it will need to access models from both apps. Therefore we've
added a dependency that specifies the last migration of ``app2``::
class Migration(migrations.Migration):
dependencies = [
('app1', '0001_initial'),
# added dependency to enable using models from app2 in move_m1
('app2', '0004_foobar'),
]
operations = [
migrations.RunPython(move_m1),
]
More advanced migrations
------------------------
If you're interested in the more advanced migration operations, or want
to be able to write your own, see the :doc:`migration operations reference
</ref/migration-operations>` and the "how-to" on :doc:`writing migrations
</howto/writing-migrations>`.
.. _migration-trimming:
Trimming migrations
===================
You are encouraged to make migrations freely and not worry about how many you
have; the migration code is optimized to deal with hundreds at a time without
much slowdown. However, eventually you will want to move back from having
several hundred migrations to just a few. That's where trimming migrations
comes in.
Trimming is the act of reducing an existing set of many migrations down to
one (or sometimes a few) migrations which still represent the same changes.
There are two common approaches to trimming migrations. The first is to use the
``squashmigrations`` command to create a merged migration file, however this
approach can be impractical on complex projects. The second approach is to
coordinate with your team to ensure that all installations of your app are up
to date, then to have a scheduled time when migrations are removed and
recreated from scratch.
Which approach is best for your project will depend on the complexity of it
and the flexibility of your team.
.. _manual-migration-trimming:
Trimming migrations manually
----------------------------
The easiest and most reliable way to trim migrations is to do so manually,
instead of using the ``squashmigrations`` command. When doing this process, any
apps that have foreign key interdependencies will need to be trimmed at the same
time so that old migrations do not depend on deleted migrations. For example,
if your ``pizza`` app has a ``ForeignKey`` to an ``Ingredients`` model in your
``food`` app, you will need to manually trim both the ``food`` and the
``pizza`` apps simultaneously. It is often easiest to trim all apps in your
project at the same time.
Once you have identified the apps you plan to trim, ensure that all of your
installations of your project are up to date with the same migrations and
ensure that no new migrations will be created until this process is complete.
To check that all migrations are in place, run the following on all
installations of your project::
$ ./manage.py makemigrations
And::
$ ./manage.py showmigrations
``makemigrations`` should show "No changes detected" and ``showmigrations``
should show "[X]" for all migrations, indicating that they have been applied.
After all of your databases are up to date, you need to clear the migration
history table. Run the following for each of your apps you plan to migrate::
$ ./manage.py migrate --fake my_app1 zero
$ ./manage.py migrate --fake my_app2 zero
...
If you run ``showmigrations`` again, it will now show that none of your
migrations have been applied.
Next, save any data or SQL migrations that you have. Find these migrations by
searching for ``RunPython`` and ``RunSQL`` in your migration files, and copy
anything you find to a temporary file.
Delete all of the migrations in the affected apps. In a Unix system the
following will generally work::
$ cd my_app1
$ find . -path "migrations/*.py" -not -name "__init__.py" -delete
Create migrations for each app. For apps with interdependencies, this may
create more than one migration::
$ ./manage.py makemigrations
For any app that had data or SQL migrations, analyze whether those migrations
are still needed. For those that are needed, recreate empty migration files::
$ ./manage.py makemigrations --empty my_app1 my_app2
Using the temporary file you made above, recreate the data and SQL migrations
that you need by copying over any old ``RunPython`` or ``RunSQL`` commands.
You should now have new migration files representing the state of your
database and any data migrations that you had prior to starting this process.
Because your database already has these changes applied by your old migrations,
you do not need to actually migrate it, but these migrations do need to be
marked as applied. Thus, fake running them::
$ ./manage.py migrate --fake
To verify that everything worked, check that the migration has been
applied successfully::
$ ./manage.py showmigrations
This computer is now complete. To deploy this to other computers running the
code base, begin by clearing the migration history on those computers::
$ ./manage.py migrate --fake my_app1 zero
$ ./manage.py migrate --fake my_app2 zero
...
Then, copy or deploy the new migrations to those computers. Finally, fake the
new migrations::
$ ./manage.py migrate --fake
And verify with::
$ ./manage.py showmigrations
.. _migration-squashing:
Squashing migrations
--------------------
Django squashes migrations by taking all of your existing migrations,
extracting their ``Operation``\s, putting them all in sequence, and then
running an optimizer over them to try and reduce the length of the list - for
example, it knows that :class:`~django.db.migrations.operations.CreateModel`
and :class:`~django.db.migrations.operations.DeleteModel` cancel each other
out, and it knows that :class:`~django.db.migrations.operations.AddField` can
be rolled into :class:`~django.db.migrations.operations.CreateModel`.
Once the operation sequence has been reduced as much as possible - the amount
possible depends on how closely intertwined your models are and if you have
any :class:`~django.db.migrations.operations.RunSQL`
or :class:`~django.db.migrations.operations.RunPython` operations (which can't
be optimized through unless they are marked as ``elidable``) - Django will then
write it back out into a new set of migration files.
These files are marked to say they replace the previously-squashed migrations,
so they can coexist with the old migration files, and Django will intelligently
switch between them depending where you are in the history. If you're still
part-way through the set of migrations that you squashed, it will keep using
them until it hits the end and then switch to the squashed history, while new
installs will use the new squashed migration and skip all the old ones.
This enables you to squash and not mess up systems currently in production
that aren't fully up-to-date yet. The recommended process is to squash, keeping
the old files, commit and release, wait until all systems are upgraded with
the new release (or if you're a third-party project, ensure your users upgrade
releases in order without skipping any), and then remove the old files, commit
and do a second release.
The command that backs all this is :djadmin:`squashmigrations` - pass it the
app label and migration name you want to squash up to, and it'll get to work::
$ ./manage.py squashmigrations myapp 0004
Will squash the following migrations:
- 0001_initial
- 0002_some_change
- 0003_another_change
- 0004_undo_something
Do you wish to proceed? [yN] y
Optimizing...
Optimized from 12 operations to 7 operations.
Created new squashed migration /home/andrew/Programs/DjangoTest/test/migrations/0001_squashed_0004_undo_somthing.py
You should commit this migration but leave the old ones in place;
the new migration will be used for new installs. Once you are sure
all instances of the codebase have applied the migrations you squashed,
you can delete them.
Use the :option:`squashmigrations --squashed-name` option if you want to set
the name of the squashed migration rather than use an autogenerated one.
Note that model interdependencies in Django can get very complex, and squashing
may result in migrations that do not run; either mis-optimized (in which case
you can try again with ``--no-optimize``, though you should also report an issue),
or with a ``CircularDependencyError``, in which case you can manually resolve it.
To manually resolve a ``CircularDependencyError``, break out one of
the ForeignKeys in the circular dependency loop into a separate
migration, and move the dependency on the other app with it. If you're unsure,
see how :djadmin:`makemigrations` deals with the problem when asked to create
brand new migrations from your models. In a future release of Django,
:djadmin:`squashmigrations` will be updated to attempt to resolve these errors
itself.
Once you've squashed your migration, you should then commit it alongside the
migrations it replaces and distribute this change to all running instances
of your application, making sure that they run ``migrate`` to store the change
in their database.
You must then transition the squashed migration to a normal migration by:
- Deleting all the migration files it replaces.
- Updating all migrations that depend on the deleted migrations to depend on
the squashed migration instead.
- Removing the ``replaces`` attribute in the ``Migration`` class of the
squashed migration (this is how Django tells that it is a squashed migration).
.. note::
Once you've squashed a migration, you should not then re-squash that squashed
migration until you have fully transitioned it to a normal migration.
.. _migration-serializing:
Serializing values
==================
Migrations are Python files containing the old definitions of your models
- thus, to write them, Django must take the current state of your models and
serialize them out into a file.
While Django can serialize most things, there are some things that we just
can't serialize out into a valid Python representation - there's no Python
standard for how a value can be turned back into code (``repr()`` only works
for basic values, and doesn't specify import paths).
Django can serialize the following:
- ``int``, ``float``, ``bool``, ``str``, ``bytes``, ``None``, ``NoneType``
- ``list``, ``set``, ``tuple``, ``dict``, ``range``.
- ``datetime.date``, ``datetime.time``, and ``datetime.datetime`` instances
(include those that are timezone-aware)
- ``decimal.Decimal`` instances
- ``enum.Enum`` instances
- ``uuid.UUID`` instances
- :func:`functools.partial` and :class:`functools.partialmethod` instances
which have serializable ``func``, ``args``, and ``keywords`` values.
- Pure and concrete path objects from :mod:`pathlib`. Concrete paths are
converted to their pure path equivalent, e.g. :class:`pathlib.PosixPath` to
:class:`pathlib.PurePosixPath`.
- :class:`os.PathLike` instances, e.g. :class:`os.DirEntry`, which are
converted to ``str`` or ``bytes`` using :func:`os.fspath`.
- ``LazyObject`` instances which wrap a serializable value.
- Enumeration types (e.g. ``TextChoices`` or ``IntegerChoices``) instances.
- Any Django field
- Any function or method reference (e.g. ``datetime.datetime.today``) (must be in module's top-level scope)
- Unbound methods used from within the class body
- Any class reference (must be in module's top-level scope)
- Anything with a custom ``deconstruct()`` method (:ref:`see below <custom-deconstruct-method>`)
.. versionchanged:: 3.2
Serialization support for pure and concrete path objects from
:mod:`pathlib`, and :class:`os.PathLike` instances was added.
Django cannot serialize:
- Nested classes
- Arbitrary class instances (e.g. ``MyClass(4.3, 5.7)``)
- Lambdas
.. _custom-migration-serializers:
Custom serializers
------------------
You can serialize other types by writing a custom serializer. For example, if
Django didn't serialize :class:`~decimal.Decimal` by default, you could do
this::
from decimal import Decimal
from django.db.migrations.serializer import BaseSerializer
from django.db.migrations.writer import MigrationWriter
class DecimalSerializer(BaseSerializer):
def serialize(self):
return repr(self.value), {'from decimal import Decimal'}
MigrationWriter.register_serializer(Decimal, DecimalSerializer)
The first argument of ``MigrationWriter.register_serializer()`` is a type or
iterable of types that should use the serializer.
The ``serialize()`` method of your serializer must return a string of how the
value should appear in migrations and a set of any imports that are needed in
the migration.
.. _custom-deconstruct-method:
Adding a ``deconstruct()`` method
---------------------------------
You can let Django serialize your own custom class instances by giving the class
a ``deconstruct()`` method. It takes no arguments, and should return a tuple
of three things ``(path, args, kwargs)``:
* ``path`` should be the Python path to the class, with the class name included
as the last part (for example, ``myapp.custom_things.MyClass``). If your
class is not available at the top level of a module it is not serializable.
* ``args`` should be a list of positional arguments to pass to your class'
``__init__`` method. Everything in this list should itself be serializable.
* ``kwargs`` should be a dict of keyword arguments to pass to your class'
``__init__`` method. Every value should itself be serializable.
.. note::
This return value is different from the ``deconstruct()`` method
:ref:`for custom fields <custom-field-deconstruct-method>` which returns a
tuple of four items.
Django will write out the value as an instantiation of your class with the
given arguments, similar to the way it writes out references to Django fields.
To prevent a new migration from being created each time
:djadmin:`makemigrations` is run, you should also add a ``__eq__()`` method to
the decorated class. This function will be called by Django's migration
framework to detect changes between states.
As long as all of the arguments to your class' constructor are themselves
serializable, you can use the ``@deconstructible`` class decorator from
``django.utils.deconstruct`` to add the ``deconstruct()`` method::
from django.utils.deconstruct import deconstructible
@deconstructible
class MyCustomClass:
def __init__(self, foo=1):
self.foo = foo
...
def __eq__(self, other):
return self.foo == other.foo
The decorator adds logic to capture and preserve the arguments on their
way into your constructor, and then returns those arguments exactly when
deconstruct() is called.
Supporting multiple Django versions
===================================
If you are the maintainer of a third-party app with models, you may need to
ship migrations that support multiple Django versions. In this case, you should
always run :djadmin:`makemigrations` **with the lowest Django version you wish
to support**.
The migrations system will maintain backwards-compatibility according to the
same policy as the rest of Django, so migration files generated on Django X.Y
should run unchanged on Django X.Y+1. The migrations system does not promise
forwards-compatibility, however. New features may be added, and migration files
generated with newer versions of Django may not work on older versions.
.. seealso::
:doc:`The Migrations Operations Reference </ref/migration-operations>`
Covers the schema operations API, special operations, and writing your
own operations.
:doc:`The Writing Migrations "how-to" </howto/writing-migrations>`
Explains how to structure and write database migrations for different
scenarios you might encounter.