Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixed #24686 -- Allow moving a model between apps. #16905

Open
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

DevilsAutumn
Copy link
Contributor

@DevilsAutumn DevilsAutumn commented May 30, 2023

Google Summer of Code 2023

Ticket: #24686
Forum: here.
Discussion: PR
Proposal: Allow moving a model between apps.

Phase 1: Focusing on operations for moving model + tests related + documentation.
Phase 2: Auto-detecting moved model, generating migrations and squashing migrations + tests related + documentation.

@DevilsAutumn
Copy link
Contributor Author

DevilsAutumn commented May 30, 2023

I think more tests are required in tests/migrations/test_operations to show that no data is lost. But how should I add so many operations in a single test?

test_move_model_with_fk
def test_move_model_with_fk(self):
        app_label_1 = "test_mmw_fk_1"
        app_label_2 = "test_mmw_fk_2"
        project_state = self.apply_operations(
            app_label_1,
            ProjectState(),
            operations=[
                migrations.CreateModel(
                    "Rider",
                    fields=[
                        ("id", models.AutoField(primary_key=True)),
                    ],
                ),
                migrations.CreateModel(
                    "TempRider",
                    fields=[
                        ("id", models.AutoField(primary_key=True)),
                    ],
                ),
            ],
        )
        project_state = self.apply_operations(
            app_label_2,
            project_state,
            operations=[
                migrations.CreateModel(
                    "Pony",
                    fields=[
                        ("id", models.AutoField(primary_key=True)),
                        ("riders", models.ForeignKey(
                            f"{app_label_1}.TempRider", on_delete=models.CASCADE)),
                    ],
                ),
            ],
        )
    self.assertTableExists(f"{app_label_1}_temprider")
    self.assertTableExists(f"{app_label_2}_pony")
    temp_rider = project_state.apps.get_model(app_label_1, "TempRider")
    pony = project_state.apps.get_model(app_label_2, "Pony")
    pony.objects.create(riders=temp_rider.objects.create())

    project_state_2 = project_state.clone()
    project_state = self.apply_operations(
        app_label_1,
        project_state,
        operations=[
            migrations.AlterModelTable(
                "TempRider",
                f"{app_label_2}_temprider",
                state_only_op=True,
            ),
        ],
    )
    project_state = self.apply_operations(
        app_label_2,
        project_state,
        operations=[
            migrations.CreateModel(
                "TempRider",
                fields=[
                    ("id", models.AutoField(primary_key=True)),
                ],
                state_only_op=True
            ),
            migrations.AlterField(
                "pony",
                name="riders",
                field=models.ForeignKey(
                    on_delete=models.CASCADE, to=f"{app_label_2}.temprider"
                ),
            ),
        ],
    )
    project_state = self.apply_operations(
        app_label_1,
        project_state,
        operations=[
            migrations.DeleteModel(
                "TempRider",
            ),
        ],
    )
    self.assertTableExists(f"{app_label_2}_temprider")
    self.assertTableExists(f"{app_label_2}_pony")
    temp_rider = project_state.apps.get_model(app_label_2, "TempRider")
    pony = project_state.apps.get_model(app_label_2, "Pony")
    pony.objects.create(riders=temp_rider.objects.create())
    self.assertEqual(temp_rider.objects.count(), 2)
    self.assertEqual(pony.objects.count(), 2)

This test is passing on sqlite3 and mysql.
Edit: I have added the above test in migrations/test_operations.py .

@DevilsAutumn DevilsAutumn marked this pull request as draft May 30, 2023 18:03
@DevilsAutumn DevilsAutumn force-pushed the ticket_#24686 branch 2 times, most recently from 3337523 to dfce113 Compare May 31, 2023 09:46
@DevilsAutumn
Copy link
Contributor Author

DevilsAutumn commented Jun 8, 2023

I have created two subclasses CreateModelInAppState and DeleteModelInAppState instead of state_only_op flag. It looks much better and less confusing now. 🤔

@felixxm
Copy link
Member

felixxm commented Jun 9, 2023

I have created two subclasses CreateModelInAppState and DeleteModelInAppState instead of state_only_op flag. It looks much better and less confusing now. thinking

Thanks 👍 . Most (all?) of new test apps won't be necessary when we will add auto-detection of moved models.

@DevilsAutumn
Copy link
Contributor Author

Most (all?) of new test apps won't be necessary when we will add auto-detection of moved models.

Should i remove all the test apps then?

@felixxm
Copy link
Member

felixxm commented Jun 9, 2023

Most (all?) of new test apps won't be necessary when we will add auto-detection of moved models.

Should i remove all the test apps then?

They can stay for now, we can remove them later when auto-detection is implemented.

@DevilsAutumn DevilsAutumn force-pushed the ticket_#24686 branch 2 times, most recently from a03687f to 4b22b17 Compare June 26, 2023 09:22
@DevilsAutumn
Copy link
Contributor Author

@felixxm i've pushed the code for autodetection of moved model .👍 Specific commit is a03687f

Copy link
Member

@felixxm felixxm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@DevilsAutumn Thanks for updates 👍 I've tried this patch on a sample project (two app, three models) and both migrate and sqlmigrate crash 😞

$ python manage.py sqlmigrate test_one 0003
Traceback (most recent call last):
  File "/django/tickets_projects/ticket_24686/manage.py", line 22, in <module>
    main()
  File "/django/tickets_projects/ticket_24686/manage.py", line 18, in main
    execute_from_command_line(sys.argv)
  File "/django/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/django/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/django/django/core/management/base.py", line 412, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/django/django/core/management/commands/sqlmigrate.py", line 38, in execute
    return super().execute(*args, **options)
  File "/django/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
  File "/django/django/core/management/commands/sqlmigrate.py", line 80, in handle
    sql_statements = loader.collect_sql(plan)
  File "/django/django/db/migrations/loader.py", line 381, in collect_sql
    state = migration.apply(state, schema_editor, collect_sql=True)
  File "/django/django/db/migrations/migration.py", line 132, in apply
    operation.database_forwards(
  File "/django/django/db/migrations/operations/fields.py", line 228, in database_forwards
    to_model = to_state.apps.get_model(app_label, self.model_name)
  File "/django/django/utils/functional.py", line 47, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/django/django/db/migrations/state.py", line 566, in apps
    return StateApps(self.real_apps, self.models)
  File "/django/django/db/migrations/state.py", line 637, in __init__
    raise ValueError("\n".join(error.msg for error in errors))
ValueError: The field test_one.MyRelatedModels.field_m2m was declared with a lazy reference to 'test_one.mymodel', but app 'test_one' doesn't provide model 'mymodel'.
The field test_one.MyRelatedModels_field_m2m.mymodel was declared with a lazy reference to 'test_one.mymodel', but app 'test_one' doesn't provide model 'mymodel'.
$ python manage.py migrate
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, sessions, test_one, test_two
Running migrations:
  Applying test_one.0001_initial... OK
  Applying test_two.0001_initial... OK
  Applying test_one.0002_alter_mymodel_options... OK
  Applying test_two.0002_mymodel_alter_myothermodel_field_fk... OK
  Applying test_one.0003_delete_mymodel_alter_myrelatedmodels_field_fk_and_more...Traceback (most recent call last):
  File "/django/tickets_projects/ticket_24686/manage.py", line 22, in <module>
    main()
  File "/django/tickets_projects/ticket_24686/manage.py", line 18, in main
    execute_from_command_line(sys.argv)
  File "/django/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/django/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/django/django/core/management/base.py", line 412, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/django/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
  File "/django/django/core/management/base.py", line 106, in wrapper
    res = handle_func(*args, **kwargs)
  File "/django/django/core/management/commands/migrate.py", line 356, in handle
    post_migrate_state = executor.migrate(
  File "/django/django/db/migrations/executor.py", line 135, in migrate
    state = self._migrate_all_forwards(
  File "/django/django/db/migrations/executor.py", line 167, in _migrate_all_forwards
    state = self.apply_migration(
  File "/django/django/db/migrations/executor.py", line 252, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/django/django/db/migrations/migration.py", line 132, in apply
    operation.database_forwards(
  File "/django/django/db/migrations/operations/fields.py", line 235, in database_forwards
    schema_editor.alter_field(from_model, from_field, to_field)
  File "/django/django/db/backends/sqlite3/schema.py", line 123, in alter_field
    if self._is_moving_model_field(old_field, new_field):
  File "/django/django/db/backends/base/schema.py", line 1617, in _is_moving_model_field
    old_app_label = old_field.remote_field.model._meta.app_label
AttributeError: 'str' object has no attribute '_meta'

It's critical to this project to run many experiments locally with various configurations of apps and models before moving forward.

django/db/backends/base/schema.py Outdated Show resolved Hide resolved
django/db/migrations/questioner.py Outdated Show resolved Hide resolved
@DevilsAutumn
Copy link
Contributor Author

By mistake i removed the record of dependency for altering model options in old app because of which the fields in old app were being altered after the DeleteModel() operation of moving model. I'm now working on a failing case - when i run makemigrations again after migrations of moving model are generated, a new migration for altering model options for moved model is generated. If I pass managed=False along with db_table in moved model in new app, no new migration is generated.

@felixxm
Copy link
Member

felixxm commented Jul 14, 2023

On SQLite, tables with relations to a moved model are rebuild (we should add a test to check performed queries and fix this), e.g.

$ python manage.py sqlmigrate test_one 0003
BEGIN;
--
-- Alter field field_fk on myrelatedmodels
--
CREATE TABLE "new__test_one_myrelatedmodels" ("id" integer NOT NULL PRIMARY KEY AUTOINCREMENT, "field_fk_id" bigint NULL REFERENCES "test_one_mymodel" ("id") DEFERRABLE INITIALLY DEFERRED);
INSERT INTO "new__test_one_myrelatedmodels" ("id", "field_fk_id") SELECT "id", "field_fk_id" FROM "test_one_myrelatedmodels";
DROP TABLE "test_one_myrelatedmodels";
ALTER TABLE "new__test_one_myrelatedmodels" RENAME TO "test_one_myrelatedmodels";
CREATE INDEX "test_one_myrelatedmodels_field_fk_id_2e7a4f46" ON "test_one_myrelatedmodels" ("field_fk_id");
--
-- Alter field field_m2m on myrelatedmodels
--
-- (no-op)
--
-- Delete model mymodel
--
-- (no-op)
COMMIT;
$ python manage.py sqlmigrate test_two 0002
BEGIN;
--
-- Create model MyModel
--
-- (no-op)
--
-- Alter field field_fk on myothermodel
--
CREATE TABLE "new__test_two_myothermodel" ("id" integer NOT NULL PRIMARY KEY AUTOINCREMENT, "field_fk_id" bigint NULL REFERENCES "test_one_mymodel" ("id") DEFERRABLE INITIALLY DEFERRED);
INSERT INTO "new__test_two_myothermodel" ("id", "field_fk_id") SELECT "id", "field_fk_id" FROM "test_two_myothermodel";
DROP TABLE "test_two_myothermodel";
ALTER TABLE "new__test_two_myothermodel" RENAME TO "test_two_myothermodel";
CREATE INDEX "test_two_myothermodel_field_fk_id_d5d44589" ON "test_two_myothermodel" ("field_fk_id");
COMMIT;

Also, please fix schema.tests.SchemaTests.test_m2m_through_remove and remove all example apps from migrations tests (they should be replaced with a autodetector/schema tests, where appropriate).

@DevilsAutumn
Copy link
Contributor Author

DevilsAutumn commented Jul 14, 2023

On SQLite, tables with relations to a moved model are rebuild (we should add a test to check performed queries and fix this),

Its because in field_fk ,null=True was added when the model was moved to other app in your project. I checked when a db attribute of a field is altered table is rebuilt in SQLite. I tested on MYSQL and the table was only altered not rebuild. Please correct me if i'm wrong.
On MYSQL:

python manage.py sqlmigrate test_one 0003
--
-- Alter field field_fk on myrelatedmodels
--
ALTER TABLE `test_one_myrelatedmodels` DROP FOREIGN KEY `test_one_myrelatedmo_field_fk_id_2e7a4f46_fk_test_one_`;
ALTER TABLE `test_one_myrelatedmodels` MODIFY `field_fk_id` bigint NULL;
ALTER TABLE `test_one_myrelatedmodels` ADD CONSTRAINT `test_one_myrelatedmo_field_fk_id_2e7a4f46_fk_test_one_` FOREIGN KEY (`field_fk_id`) REFERENCES `test_one_mymodel` (`id`);
--
-- Alter field field_m2m on myrelatedmodels
--
-- (no-op)
--
-- Delete model mymodel
--
-- (no-op)
python manage.py sqlmigrate test_two 0002
--
-- Create model MyModel
--
-- (no-op)
--
-- Alter field field_fk on myothermodel
--
ALTER TABLE `test_two_myothermodel` DROP FOREIGN KEY `test_two_myothermode_field_fk_id_d5d44589_fk_test_one_`;
ALTER TABLE `test_two_myothermodel` MODIFY `field_fk_id` bigint NULL;
ALTER TABLE `test_two_myothermodel` ADD CONSTRAINT `test_two_myothermode_field_fk_id_d5d44589_fk_test_one_` FOREIGN KEY (`field_fk_id`) REFERENCES `test_one_mymodel` (`id`);

Should i still add a test to check performed queries?

@felixxm
Copy link
Member

felixxm commented Jul 15, 2023

Its because in field_fk ,null=True was added when the model was moved to other app in your project. I checked when a db attribute of a field is altered table is rebuilt in SQLite. I tested on MYSQL and the table was only altered not rebuild. Please correct me if i'm wrong.

Yes, you are right.

@DevilsAutumn
Copy link
Contributor Author

I was testing this patch for different use cases and I came across a scenario with potential error:

Consider 3 models ,A, B and C. Model C has fk to A and B, and B has m2m field to A through C. We move model B to app_two first and then move model C to app_two. For moving model C we have to change the through of model B to test_two.C and hence it should have an AlterField operation which is missing.

If we only move Model C to app_two and not B , AlterField is added for the m2m field as through is changed.
Will update the PR as soon as it is solved.

Copy link
Member

@shaib shaib left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I understand correctly, the current approach is to set the models (both the removed and the added one) to managed=False so that the operations performed on them are not applied in the database.

If this has other advantages over the use of SeparateDatabaseAndState, then it may be justified, however, as the code stands -- if I understand correctly -- the added model is left unmanaged in the end (that is, take model A from app first to app second, and now -- in the migrations -- second.A is unmanaged). The next makemigrations will create a migration to change it back to managed, which will surprise the user.

In general, the use of managed here feels wrong, even if the code is simpler in the implementation and shorter in the generated migration -- because it introduces an unrelated concept into the generated migration code.

django/db/migrations/autodetector.py Show resolved Hide resolved
django/db/migrations/autodetector.py Outdated Show resolved Hide resolved
django/db/migrations/autodetector.py Outdated Show resolved Hide resolved
django/db/migrations/autodetector.py Outdated Show resolved Hide resolved
django/db/migrations/autodetector.py Outdated Show resolved Hide resolved
django/db/migrations/autodetector.py Outdated Show resolved Hide resolved
@DevilsAutumn
Copy link
Contributor Author

DevilsAutumn commented Jul 28, 2023

@shaib Thankyou for reviewing! I'll push the suggested changes shortly.

If this has other advantages over the use of SeparateDatabaseAndState, then it may be justified, however, as the code stands -- if I understand correctly -- the added model is left unmanaged in the end (that is, take model A from app first to app second, and now -- in the migrations -- second.A is unmanaged). The next makemigrations will create a migration to change it back to managed, which will surprise the user.

If we're marking the model managed, then after the migration for moving a model are created we have to add managed=False in the model's Meta for now, so that it doesn't create new migration to change it back. Working to avoid creating new migration .

@charettes
Copy link
Member

charettes commented Aug 31, 2023

I wonder if it is really required ? 🤔 because un-managed model creation migration running before optional rename + marking model unmanaged migration shouldn't cause any trouble .

I believe it is at least as otherwise any usage of the un-managed models in the moved-to application will crash if the moved-from managed model table doesn't exist or has the wrong table name.

Keep in mind that there's no way to guarantee that a migration runs immediately after another, any other migration with similar requirement could be interleaved and try to use the moved-to unmanaged models so we must ensure that models are always in a usable state at migrations boundaries.

@DevilsAutumn
Copy link
Contributor Author

un-managed model creation migration can't depend on the optional rename + marking model unmanaged migration both. If we did so, the next migrations will be optimized and the model will be created as managed(due to AlterModelOptions optimization) --> and then DeleteModel in old_app .

And if we modify the code to run AlterFields (if any) + DeleteModel in old_app --> AlterModelOptions + AlterField (if any) in new_app after un-managed model creation migration, then Altering m2m field in last migration in new app will give error since old from model is deleted. 🤕

But it might work if un-managed model creation migration depends only on the optional rename migration and rest remains the same. 🤔

Copy link
Member

@felixxm felixxm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I squashed commits, rebased, and adjusted to recent changes (7dd3e69). I think it's ready for another round of reviews.

@felixxm felixxm force-pushed the ticket_#24686 branch 2 times, most recently from 2c92e98 to ad15d7b Compare November 21, 2023 10:08
@DevilsAutumn
Copy link
Contributor Author

cool 👍

Copy link
Member

@David-Wobrock David-Wobrock left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't checkout the code locally, but from what I read, this looks pretty sweet 👍
Good job!

Copy link
Member

@shaib shaib left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added some notes on the documentation. This seems to be getting really close.

docs/releases/5.1.txt Outdated Show resolved Hide resolved
docs/releases/5.1.txt Outdated Show resolved Hide resolved
docs/topics/migrations.txt Outdated Show resolved Hide resolved
docs/topics/migrations.txt Outdated Show resolved Hide resolved
docs/topics/migrations.txt Outdated Show resolved Hide resolved
unless the moved model in new app defines a ``Meta.db_table`` that matches
the old one.

To move a model (without a table rename), we first create the new model, making
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This paragraph explains the migration operations generated by the framework for a model move, and I think this should be said very explicitly, because as the text stands, it can be understood to describe the operations the user needs to take. So I would change the text here along these lines -- clearly separate and describe what the user needs to do, and what the framework does.

Something like:
"""

To move a model from one app to another, move the model's code (the class definition) from the old app's models.py file to the new app's models.py, and run the makemigrations management command. This command will generate migrations in both apps, with the following operations:

  • First, create the new model, making it point to the already existing table. To make Django not attempt to create that table (which would fail), mark it managed=False upon creation (after this operation, we momentarily have two models pointing at the same table).
  • Then, to make it possible to remove the old model without deleting the table, mark it managed=False as well.
  • Finally, remove the old model and mark the new one managed=True (these are two migrations, as each is in its own app)

Note that this only handles the migrations part -- you will still need to handle other aspects of the move (such as changing all existing references to the model to point to its new location) yourself (in fact, you will probably need to handle these before running makemigrations, for Django to be able to load the apps).
"""

(the above is my style of writing, Django documentation usually prefers not to use parentheses so liberally)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks 👍 this looks better. Also should i move the last para under a separate note? It'll look something like this:

Screenshot 2024-02-06 at 9 02 16 PM

@bachi76
Copy link

bachi76 commented Mar 23, 2024

We really need this. Just sayin' :)

@felixxm felixxm removed their assignment Mar 26, 2024
@sarahboyce sarahboyce changed the title [WIP]Ticket #24686 -- Allow moving a model between apps. Fixed #24686 -- Allow moving a model between apps. Apr 4, 2024
@sarahboyce
Copy link
Contributor

Was able to do a simple case on SQLite and it moved very well 🥳 experience felt magical

I think I have found an issue when moving on SQLite in the multitable inheritance case.

Starting point

Migrations applied and all tables have some data

app1/models.py
from django.db import models


class Base(models.Model):
    data = models.TextField(default="Example")


class ExtendBase(Base):
    field_data = models.TextField(default="Field data")

Moving

app1/models.py
from django.db import models
from app2.models import Base


class ExtendBase(Base):
    field_data = models.TextField(default="Field data")
app2/models.py
from django.db import models


class Base(models.Model):
    data = models.TextField(default="Example")

I run: python manage.py makemigrations

console
Was the model app1.Base moved to app2.Base? [y/N] y
Migrations for 'app1':
  app1\migrations\0006_alter_base_table.py
    ~ Rename table for base to app2_base
  app1\migrations\0007_alter_extendbase_base_ptr_alter_base_options.py
    ~ Alter field base_ptr on extendbase
    ~ Change Meta options on base
  app1\migrations\0008_delete_base.py
    - Delete model base
Migrations for 'app2':
  app2\migrations\0003_base.py
    + Create model base
  app2\migrations\0004_alter_base_options.py
    ~ Change Meta options on base

Created migration files

app1\migrations\0006_alter_base_table.py
# Generated by Django 5.1 on 2024-04-19 13:10

from django.db import migrations


class Migration(migrations.Migration):

    dependencies = [
        ('app1', '0005_delete_mysimplemodel'),
    ]

    operations = [
        migrations.AlterModelTable(
            name='base',
            table='app2_base',
        ),
    ]
app1\migrations\0007_alter_extendbase_base_ptr_alter_base_options.py
# Generated by Django 5.1 on 2024-04-19 13:10

import django.db.models.deletion
from django.db import migrations, models


class Migration(migrations.Migration):

    dependencies = [
        ('app1', '0006_alter_base_table'),
        ('app2', '0003_base'),
    ]

    operations = [
        migrations.AlterField(
            model_name='extendbase',
            name='base_ptr',
            field=models.OneToOneField(auto_created=True, on_delete=django.db.models.deletion.CASCADE, parent_link=True, primary_key=True, serialize=False, to='app2.base'),
        ),
        migrations.AlterModelOptions(
            name='base',
            options={'managed': False},
        ),
    ]
app1\migrations\0008_delete_base.py
# Generated by Django 5.1 on 2024-04-19 13:10

from django.db import migrations


class Migration(migrations.Migration):

    dependencies = [
        ('app1', '0007_alter_extendbase_base_ptr_alter_base_options'),
        ('app2', '0004_alter_base_options'),
    ]

    operations = [
        migrations.DeleteModel(
            name='base',
        ),
    ]
app2\migrations\0003_base.py
# Generated by Django 5.1 on 2024-04-19 13:10

from django.db import migrations, models


class Migration(migrations.Migration):

    dependencies = [
        ('app1', '0006_alter_base_table'),
        ('app2', '0002_initial'),
    ]

    operations = [
        migrations.CreateModel(
            name='base',
            fields=[
                ('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
                ('data', models.TextField(default='Example')),
            ],
            options={
                'indexes': [],
                'constraints': [],
                'managed': False,
            },
        ),
    ]
app2\migrations\0004_alter_base_options.py
# Generated by Django 5.1 on 2024-04-19 13:10

from django.db import migrations


class Migration(migrations.Migration):

    dependencies = [
        ('app1', '0007_alter_extendbase_base_ptr_alter_base_options'),
        ('app2', '0003_base'),
    ]

    operations = [
        migrations.AlterModelOptions(
            name='base',
            options={},
        ),
    ]

When migrating it crashed with the following error:

Operations to perform:
  Apply all migrations: admin, app1, app2, auth, contenttypes, flatpages, sessions, sites
Running migrations:
  Applying app1.0006_alter_base_table... OK
  Applying app2.0003_base... OK
  Applying app1.0007_alter_extendbase_base_ptr_alter_base_options...Traceback (most recent call last):
  File "path\django-move-model-between-apps\mysite\manage.py", line 22, in <module>
    main()
  File "path\django-move-model-between-apps\mysite\manage.py", line 18, in main
    execute_from_command_line(sys.argv)
  File "path\django-move-model-between-apps\venv\Lib\site-packages\django\core\management\__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "path\django-move-model-between-apps\venv\Lib\site-packages\django\core\management\__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "path\django-move-model-between-apps\venv\Lib\site-packages\django\core\management\base.py", line 413, in run_from_argv
    self.execute(*args, **cmd_options)
  File "path\django-move-model-between-apps\venv\Lib\site-packages\django\core\management\base.py", line 459, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "path\django-move-model-between-apps\venv\Lib\site-packages\django\core\management\base.py", line 107, in wrapper
    res = handle_func(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "path\django-move-model-between-apps\venv\Lib\site-packages\django\core\management\commands\migrate.py", line 356, in handle
  File "path\django-move-model-between-apps\venv\Lib\site-packages\django\db\migrations\state.py", line 664, in render_multiple
    model.render(self)
  File "path\django-move-model-between-apps\venv\Lib\site-packages\django\db\migrations\state.py", line 957, in render
    return type(self.name, bases, body)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\boyce\PycharmProjects\django-move-model-between-apps\venv\Lib\site-packages\django\db\models\base.py", line 291, in __new__
    raise FieldError(
django.core.exceptions.FieldError: Auto-generated field 'base_ptr' in class 'ExtendBase' for parent_link to base class 'Base' clashes with declared field of the same name.

Can you confirm that you can replicate?

@DevilsAutumn
Copy link
Contributor Author

@sarahboyce I was able to reproduce this issue, thanks. Looking into it.

@DevilsAutumn
Copy link
Contributor Author

This is happening because when we are moving base model, the base value(model_state.bases) of extendbase model is not getting updating and still pointing to app1.base. And when AlterField operation is reloading the extendbase model after updating, its seeing it as another one to one field with same name as base_ptr. Even after we solve this issue by comparing app_label along with base pointer name here, it will still throw InvalidBasesError error because base is still pointing to app1.base but new base is app2.base.

There are two simple doc solutions for this (keeping the complications in mind) :

  1. Either we can document to not move a model which is being inherited by another model.
  2. Or we can document if someone want to move a base model inherited by other models, they'll have to move all the models inheriting that base model along with base model (like moving base and extendbase together in above case).

One complicated solution can be to create a new operation AlterModelBase and change the base of model inheriting another model while moving it (not sure of the implementation part yet).

CCing @charettes to know his views on this.

@sarahboyce
Copy link
Contributor

@DevilsAutumn I have found something else around Django's contenttypes framework that needs investigating and a proposed solution

Roughly the issue is that ContentTypes stores the app_label and model name of all models. This looks a bit like this:

>>> from django.contrib.contenttypes.models import ContentType
>>> ContentType.objects.values()
<QuerySet [{'id': 1, 'app_label': 'admin', 'model': 'logentry'}, {'id': 2, 'app_label': 'auth', 'model': 'permission'}, {'id': 3, 'app_label': 'auth', 'model': 'group'}, {'id': 4, 'app_label': 'auth', 'model': 'user'}, {'id': 5, 'app_label': 'contenttypes', 'model': 'contenttype'}, {'id': 6, 'app_label': 'sessions', 'model': 'session'}, {'id': 7, 'app_label': 'sites', 'model': 'site'}, {'id': 8, 'app_label': 'flatpages', 'model': 'flatpage'},{'id': 9, 'app_label': 'app1', 'model': 'mysimplemodel'}, {'id': 10, 'app_label': 'app1', 'model': 'taggeditem'}]>

When I move MySimpleModel from app1 to app2, makemigrations and migrate, instead of the ContentType instance {'id': 9, 'app_label': 'app1', 'model': 'mysimplemodel'} being updated to {'id': 9, 'app_label': 'app2', 'model': 'mysimplemodel'}, a new instance is created {'id': 11, 'app_label': 'app2', 'model': 'mysimplemodel'}.
This means any GenericForiegnKey using MySimpleModel breaks as they still expect it to be in app1 (as the FK is pointing at the old one).

error if useful
>>> TaggedItem.objects.get(id=1)
<TaggedItem: test>
>>> test = TaggedItem.objects.get(id=1)
>>> test.content_object
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "\django-move-model-between-apps\venv\Lib\site-packages\django\contrib\contenttypes\fields.py", line 261, in __get__
    rel_obj = ct.get_object_for_this_type(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "\django-move-model-between-apps\venv\Lib\site-packages\django\contrib\contenttypes\models.py", line 184, in get_object_for_this_type
    return self.model_class()._base_manager.using(using).get(**kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute '_base_manager'

I am using the model that is used in the docs example

I think this needs a test and a fix 🤔

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
7 participants