properly reset databases on tests teardown

[skarzi]

Currently multiple databases are supported by creating instance of Migrator for each database, but only default database is cleaned at the end of test - https://github.com/wemake-services/django-test-migrations/blob/master/django_test_migrations/migrator.py#L77

Also using migrate management command is not the best idea when it comes to performance, because there is no need to apply all unapplied migrations just before the next test, which should start with clean state.
Django's TestCase uses flush management command to clean databases after test.

We need to reuse Django's code responsible for cleaning database on test teardown (or establish our own way to do that) and fix Migrator.reset() method to clean Migrator.database instead of default one.

[skarzi]

I've investigated and TransactionTestCase._post_teardown() is called in both cases:

• in MigratorTestCase, because its sublcass of TransactionTestCase and doesn't override any method that shadow this method call
• in migrator_factory, because it uses transactional_db fixture which uses _django_db_fixture_helper that finally add _post_teardown to request's finalizers.

So in these cases calling migrator.reset() is not necessary, because there is no point in running flush command 2 times on test's teardown.

We can refactor a bit and leave Migrator.reset() to allow developers use this class as they wish and just do not call .reset() in MigratorTestCase.tearDown() and do not add it to migrator_factory request's finalizers.

What's your opinion?

[sobolevn]

Sounds good!

[chel-ou]

I face an issue linked to this issue. The change proposed breaks my tests when using Django unittest native runner with keepdb and parallel: python manage.py test --parallel --keepdb

If I run the three commands in a row:

python manage.py test --parallel --noinput
python manage.py test --parallel --keepdb
python manage.py test --parallel --keepdb

At the last run I get errors linked to DB schema not being migrated back to the last migrations.
Running the tests with --keepdb but without --parallel do not create any issue.

I tried to understand why the situation is different between normal and parallel test runner, but couldn't find a reasonable explanation.

I tried using the new reset() method from Migrator (with flush) but it did not fix the issue.

The issue is solved by adding the following tearDown in the MigratorTestCase

def tearDown(self):
        call_command('migrate', verbosity=0) 

Any idea for an explanation or a better fix?

[skarzi]

Hello ;)

It would be great if you can share some minimal version of code that causes this issue.
If you can't no worry I will do my best to provide you some explanation and fix this issue.
Could you share following information?

1. Do you have any other tests accessing DB or the only tests you have are migrations' tests?
2. Which version of Django do you use?
3. Please paste the whole traceback you get

I think the issue may be caused by the way flush works- it doesn't TRUNCATE django_migrations, so for example let's assume following situation:
in first run of tests (with --keepdb --parallel flags) process with number n runs MigrationTestCase as the last test in its testsuite, which implies that django_migrations table is not cleared in test database with suffix n. Then in second tests run with the same flags, process n takes DB with suffix n and runs some Django's TestCase which causes error you spotted

But to confirm my thesis and provide fix, firstly please share above requested information

[chel-ou]

Thanks for your feedback. I'll work on trying to get a simple failing test to work on.

To answer your questions:

1. I have other tests running during in the test suite accessing the DB. The migration tests actually do not fail.
2. Django 2.2.10
3. One of the traceback below

======================================================================
ERROR: test_version_auto_increment (Event.tests.test_models.EventModelCleanTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedColumn: column participant_entity.description does not exist
LINE 1: ...nt_entity"."phone", "participant_entity"."name", "participa...
                                                             ^


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/unittest/case.py", line 59, in testPartExecutor
    yield
  File "/usr/local/lib/python3.7/unittest/case.py", line 628, in run
    testMethod()
  File "/code/event/tests/test_models.py", line 50, in test_version_auto_increment
    event = EventFactory()
  File "/usr/local/lib/python3.7/site-packages/factory/base.py", line 46, in __call__
    return cls.create(**kwargs)
  File "/usr/local/lib/python3.7/site-packages/factory/base.py", line 564, in create
    return cls._generate(enums.CREATE_STRATEGY, kwargs)
  File "/usr/local/lib/python3.7/site-packages/factory/django.py", line 141, in _generate
    return super(DjangoModelFactory, cls)._generate(strategy, params)
  File "/usr/local/lib/python3.7/site-packages/factory/base.py", line 501, in _generate
    return step.build()
  File "/usr/local/lib/python3.7/site-packages/factory/builder.py", line 272, in build
    step.resolve(pre)
  File "/usr/local/lib/python3.7/site-packages/factory/builder.py", line 221, in resolve
    self.attributes[field_name] = getattr(self.stub, field_name)
  File "/usr/local/lib/python3.7/site-packages/factory/builder.py", line 375, in __getattr__
    extra=context,
  File "/usr/local/lib/python3.7/site-packages/factory/declarations.py", line 321, in evaluate
    return self.generate(step, defaults)
  File "/usr/local/lib/python3.7/site-packages/factory/declarations.py", line 411, in generate
    return step.recurse(subfactory, params, force_sequence=force_sequence)
  File "/usr/local/lib/python3.7/site-packages/factory/builder.py", line 233, in recurse
    return builder.build(parent_step=self, force_sequence=force_sequence)
  File "/usr/local/lib/python3.7/site-packages/factory/builder.py", line 272, in build
    step.resolve(pre)
  File "/usr/local/lib/python3.7/site-packages/factory/builder.py", line 221, in resolve
    self.attributes[field_name] = getattr(self.stub, field_name)
  File "/usr/local/lib/python3.7/site-packages/factory/builder.py", line 375, in __getattr__
    extra=context,
  File "/usr/local/lib/python3.7/site-packages/factory/declarations.py", line 321, in evaluate
    return self.generate(step, defaults)
  File "/usr/local/lib/python3.7/site-packages/factory/declarations.py", line 411, in generate
    return step.recurse(subfactory, params, force_sequence=force_sequence)
  File "/usr/local/lib/python3.7/site-packages/factory/builder.py", line 233, in recurse
    return builder.build(parent_step=self, force_sequence=force_sequence)
  File "/usr/local/lib/python3.7/site-packages/factory/builder.py", line 272, in build
    step.resolve(pre)
  File "/usr/local/lib/python3.7/site-packages/factory/builder.py", line 221, in resolve
    self.attributes[field_name] = getattr(self.stub, field_name)
  File "/usr/local/lib/python3.7/site-packages/factory/builder.py", line 375, in __getattr__
    extra=context,
  File "/usr/local/lib/python3.7/site-packages/factory/declarations.py", line 321, in evaluate
    return self.generate(step, defaults)
  File "/usr/local/lib/python3.7/site-packages/factory/declarations.py", line 411, in generate
    return step.recurse(subfactory, params, force_sequence=force_sequence)
  File "/usr/local/lib/python3.7/site-packages/factory/builder.py", line 233, in recurse
    return builder.build(parent_step=self, force_sequence=force_sequence)
  File "/usr/local/lib/python3.7/site-packages/factory/builder.py", line 279, in build
    kwargs=kwargs,
  File "/usr/local/lib/python3.7/site-packages/factory/base.py", line 315, in instantiate
    return self.factory._create(model, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/factory/django.py", line 182, in _create
    return cls._get_or_create(model_class, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/factory/django.py", line 164, in _get_or_create
    instance, _created = manager.get_or_create(*args, **key_fields)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 538, in get_or_create
    return self.get(**kwargs), False
  File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 402, in get
    num = len(clone)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 256, in __len__
    self._fetch_all()
  File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 1242, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 55, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/sql/compiler.py", line 1133, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.7/site-packages/django/db/utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: column participant_entity.description does not exist
LINE 1: ...nt_entity"."phone", "participant_entity"."name", "participa...

[skarzi]

Thank you for posting more details.
I have investigated this issue more deeply and it seems that my thesis is true.
Currently I see following solutions:

1. Revert commit that introduce this regression - 55a5261

2. Leave flush usage, but additionally manually flush all rows from django_migrations except these related to initial migrations - it's necessary because flush only empties tables, so we need initial migrations to be present to avoid exceptions like table "app_model" already exists

3. Implement custom DB resetting that works in following way:

   a) instead of flushing Django's models tables, drop them (DROP TABLE)
   b) flush django_migrations table - initial migrations are not needed now, because we have dropped tables

3rd option is my favorite one - it is the closest to real life migrate command executing on production or some other environment, because every migration's test will start from empty database, then calling .before() will populate our production-like state and (which is important) it will do that only forward (in current implementation or this from point 1 calling .before() can migrate backwards resulting in some migration with reverse code etc being applied which wasn't expected for particular test) and finally we can migrate to migrations being tested via after().

It's really catchy topic, so if something is not clear for you, let me know and I will try to explain it better.

@sobolevn what's your thoughts on that?

[chel-ou]

Thank you @skarzi for looking into the issue.

I have been able to narrow down the issue. It actually affects the tests ran on Postgres, but not on SQLite.

I managed to make a simple failing app. I'll send it to you tomorrow.

[chel-ou]

I just uploaded the failing app in the branch below.
https://github.com/chel-ou/django-test-migrations/tree/failing_app_parallel_keepdb/django_test_failing_app

To reproduce the error, run in an environment with Postgres and django_test_migrations master the following commands:

python manage.py test --parallel 2 --noinput
python manage.py test --parallel 2 --keepdb
python manage.py test --parallel 2 --keepdb

As mentioned before, adding the following tearDown method is a fix to the issue:

def tearDown(self):
        call_command('migrate', verbosity=0) 

Below is the full output of the last command:

Using existing test database for alias 'default'...
Using existing clone for alias 'default'...
Using existing clone for alias 'default'...
System check identified no issues (0 silenced).
E.EE......
======================================================================
ERROR: test_full_clean (participant.tests.test_factories.EntityFactoryTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedColumn: column participant_entity.description does not exist
LINE 1: ...nt_entity"."hashid", "participant_entity"."name", "participa...
                                                             ^


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/unittest/case.py", line 59, in testPartExecutor
    yield
  File "/usr/local/lib/python3.7/unittest/case.py", line 628, in run
    testMethod()
  File "/code/django_test_failing_app/participant/tests/test_factories.py", line 11, in test_full_clean
    EntityFactory().full_clean()
  File "/usr/local/lib/python3.7/site-packages/factory/base.py", line 46, in __call__
    return cls.create(**kwargs)
  File "/usr/local/lib/python3.7/site-packages/factory/base.py", line 564, in create
    return cls._generate(enums.CREATE_STRATEGY, kwargs)
  File "/usr/local/lib/python3.7/site-packages/factory/django.py", line 141, in _generate
    return super(DjangoModelFactory, cls)._generate(strategy, params)
  File "/usr/local/lib/python3.7/site-packages/factory/base.py", line 501, in _generate
    return step.build()
  File "/usr/local/lib/python3.7/site-packages/factory/builder.py", line 279, in build
    kwargs=kwargs,
  File "/usr/local/lib/python3.7/site-packages/factory/base.py", line 315, in instantiate
    return self.factory._create(model, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/factory/django.py", line 182, in _create
    return cls._get_or_create(model_class, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/factory/django.py", line 164, in _get_or_create
    instance, _created = manager.get_or_create(*args, **key_fields)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 538, in get_or_create
    return self.get(**kwargs), False
  File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 402, in get
    num = len(clone)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 256, in __len__
    self._fetch_all()
  File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 1242, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 55, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/sql/compiler.py", line 1133, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.7/site-packages/django/db/utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: column participant_entity.description does not exist
LINE 1: ...nt_entity"."hashid", "participant_entity"."name", "participa...
                                                             ^


======================================================================
ERROR: test_entity_object_string (participant.tests.test_models.EntityModelTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedColumn: column participant_entity.description does not exist
LINE 1: ...nt_entity"."hashid", "participant_entity"."name", "participa...
                                                             ^


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/unittest/case.py", line 59, in testPartExecutor
    yield
  File "/usr/local/lib/python3.7/unittest/case.py", line 628, in run
    testMethod()
  File "/code/django_test_failing_app/participant/tests/test_models.py", line 16, in test_entity_object_string
    entity = EntityFactory(name='my entity')
  File "/usr/local/lib/python3.7/site-packages/factory/base.py", line 46, in __call__
    return cls.create(**kwargs)
  File "/usr/local/lib/python3.7/site-packages/factory/base.py", line 564, in create
    return cls._generate(enums.CREATE_STRATEGY, kwargs)
  File "/usr/local/lib/python3.7/site-packages/factory/django.py", line 141, in _generate
    return super(DjangoModelFactory, cls)._generate(strategy, params)
  File "/usr/local/lib/python3.7/site-packages/factory/base.py", line 501, in _generate
    return step.build()
  File "/usr/local/lib/python3.7/site-packages/factory/builder.py", line 279, in build
    kwargs=kwargs,
  File "/usr/local/lib/python3.7/site-packages/factory/base.py", line 315, in instantiate
    return self.factory._create(model, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/factory/django.py", line 182, in _create
    return cls._get_or_create(model_class, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/factory/django.py", line 164, in _get_or_create
    instance, _created = manager.get_or_create(*args, **key_fields)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 538, in get_or_create
    return self.get(**kwargs), False
  File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 402, in get
    num = len(clone)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 256, in __len__
    self._fetch_all()
  File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 1242, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 55, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/sql/compiler.py", line 1133, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.7/site-packages/django/db/utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: column participant_entity.description does not exist
LINE 1: ...nt_entity"."hashid", "participant_entity"."name", "participa...
                                                             ^


======================================================================
ERROR: test_name_cannot_be_blank (participant.tests.test_models.EntityModelTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedColumn: column participant_entity.description does not exist
LINE 1: ...nt_entity"."hashid", "participant_entity"."name", "participa...
                                                             ^


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/unittest/case.py", line 59, in testPartExecutor
    yield
  File "/usr/local/lib/python3.7/unittest/case.py", line 628, in run
    testMethod()
  File "/code/django_test_failing_app/participant/tests/test_models.py", line 13, in test_name_cannot_be_blank
    EntityFactory(name='').full_clean()
  File "/usr/local/lib/python3.7/site-packages/factory/base.py", line 46, in __call__
    return cls.create(**kwargs)
  File "/usr/local/lib/python3.7/site-packages/factory/base.py", line 564, in create
    return cls._generate(enums.CREATE_STRATEGY, kwargs)
  File "/usr/local/lib/python3.7/site-packages/factory/django.py", line 141, in _generate
    return super(DjangoModelFactory, cls)._generate(strategy, params)
  File "/usr/local/lib/python3.7/site-packages/factory/base.py", line 501, in _generate
    return step.build()
  File "/usr/local/lib/python3.7/site-packages/factory/builder.py", line 279, in build
    kwargs=kwargs,
  File "/usr/local/lib/python3.7/site-packages/factory/base.py", line 315, in instantiate
    return self.factory._create(model, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/factory/django.py", line 182, in _create
    return cls._get_or_create(model_class, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/factory/django.py", line 164, in _get_or_create
    instance, _created = manager.get_or_create(*args, **key_fields)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 538, in get_or_create
    return self.get(**kwargs), False
  File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 402, in get
    num = len(clone)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 256, in __len__
    self._fetch_all()
  File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 1242, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/usr/local/lib/python3.7/site-packages/django/db/models/query.py", line 55, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
  File "/usr/local/lib/python3.7/site-packages/django/db/models/sql/compiler.py", line 1133, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python3.7/site-packages/django/db/utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.7/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: column participant_entity.description does not exist
LINE 1: ...nt_entity"."hashid", "participant_entity"."name", "participa...
                                                             ^


----------------------------------------------------------------------
Ran 10 tests in 0.519s

FAILED (errors=3)
Preserving test database for alias 'default'...
Preserving test database for alias 'default'...
Preserving test database for alias 'default'...

[skarzi]

Thank you @chel-ou for uploading failing app ;)

So this error is related to the fact how flush command (I have explained it in #38 (comment)) and --keepdb option works, namely it apply migrations only when database is not present, otherwise it just take existing DB and use it, without any preparation etc.

I have also partially implemented (partially, because DROP TABLE should be ran with CASCADE if it's onl y supported by DB engine otherwise all constraint should be deleted before dropping tables) solution 3 from previously mentioned comment, however it doesn't solve issue raised by @chel-ou.

So now I think we should do one of following:

1. Revert commit that introduce this regression - 55a5261 - I am not fully satisfied with this solution mostly because of running backward migrations, so this is not real life way/process of doing migration on production
2. Finish update database cleaning after migrations test #46 and introduce some flag to pytest plugin and Django's test runner and run migrations test separately only if this flag passed - this doesn't solve problem when tests with --keepdb and before mentioned flag are ran firstly and next tests with only --keepdb are run in this case second tests ran will fail.
3. Apply changes from point 1 and modify Migrator .before() so it will be doing what now .reset() do in and update database cleaning after migrations test #46 before doing actual migration to migrate_from migration - this solution is probably most complicated when it comes to implementation, complexity and amount of DB operations, but it will make tests the most similar to real life process of applying migrations and should fix issues discussed above.

#46 already contains implementation of point 3

What's your opinion guys?

[sobolevn]

I don't really have an opinion on this one, because it is quite a hard topic to follow.
I trust you guys here 🙂

[chel-ou]

I also do not have a clear opinion on the topic. I have some thoughts though.

One thing that troubles me that is this bug only affects Postgres, and not SQLite. So the issue might not be at the SQL commands level, but more linked to the intricacies of implementation of Postgres.

Regarding your point 1., I am not sure I understand the philosophical problem with applying migrate after the test is completed. Migrate is a built-in command in Django, enabling us to make sure that the DB is restored to an up to date state. It may not be the most efficient method but at least we ensure compatibility with future versions of Django on this specific topic.

One (not very clean) option could be to have a different behavior in the unittest case if --keepdb and --parallel are passed as args. It is not nice, but avoids to increase the overhead for all other cases.

Regarding your points 2. and 3. I am not sure how it would solve the issue. My feeling is that it would create too much complexity compared to the issue.

[skarzi]

One thing that troubles me that is this bug only affects Postgres, and not SQLite. So the issue might not be at the SQL commands level, but more linked to the intricacies of implementation of Postgres.

I don't think it's related to DB backend.
It works on sqlite3 because Django by default (when you do not set TEST settings for sqlite3 DB) uses in-memory sqlite3 database:

When using SQLite, the tests will use an in-memory database by default (i.e., the database will be created in memory, bypassing the filesystem entirely!)

Reference

Data migrations or any complicated migrations are fragile operations, because they are ran on production database, so if anything go wrong then really bad things will happen.
It's good enough argument that such migrations must be tested really carefully and to test them carefully we need to provide environment as similar to production environment as possible. When you run migrations on production you just migrate forward (of course except some extreme cases when you need revert some changes, but such cases can be also tested by our framework), so the same should be done when you are preparing database to migrations' tests, that's the reason I think point 3 from #38 (comment) is the best solution.

[chel-ou]

Thanks for your explanations.
I read your brilliant code in #46 and I think I better understand your point.
Your issue was not so much about using migrate to restore the database in the reset phase, but using it in the preparation phase.

I confirm #46 fixes the regression.

[skarzi]

Thank you @chel-ou 👍

I would be very grateful if more people will give an opinion about PR #46

[asfaltboy]

Hi @skarzi thanks for investing much time into this, this is indeed a tough issue to solve.

I was thinking about this a bit more, and it seems that, typically, the reason data migrations fail to be re-applied in teardown is due to "primary key sequence" mismatch.
I see that in your PR #46 you implement get_django_migrations_table_sequences to keep track of the ids ...
do you think it's possible to simplify this by using django.test.TransactionTestCase.reset_sequences (also supported in pytest-django)?

Note that I tried testing #46 locally and I'm having issues with dependency migrations not being applied (while they are being applied in current 0.2.0 ). I was not able to overcome this by specifying all the related apps migrations (I gave up mid-way), but even if I did it would not be ideal. How come this is not the same behavior as on 0.2.0, I thought #46 only fixes the teardown?

[sobolevn]

@asfaltboy awesome idea! Thanks!

[skarzi]

Hello @asfaltboy,
thanks for feedback and investigation 👍

I was thinking about this a bit more, and it seems that, typically, the reason data migrations fail to be re-applied in teardown is due to "primary key sequence" mismatch

Do you mean "primary key sequence" mismatch of all models' tables or just django_migrations table?

I see that in your PR #46 you implement get_django_migrations_table_sequences to keep track of the ids ...
do you think it's possible to simplify this by using django.test.TransactionTestCase.reset_sequences (also supported in pytest-django)?

Unfortunately, it's not possible to reset sequences of django_migrations table by using this feature, because django_migrations is not "regular" Django model, so its not returned by router.get_migratable_models() which is used to implement few important things specifically connection.introspection.sequences_list() that is used to get all sequences in TransactionTestCase._reset_sequences().
So we need to keep this function if we want to reset django_migrations table sequences, however we can use reset_sequences feature to reset models' tables sequences.

Note that I tried testing #46 locally and I'm having issues with dependency migrations not being applied (while they are being applied in current 0.2.0 ).

I will try to reproduce this error and investigate it more deeply on the weekend.

I was not able to overcome this by specifying all the related apps migrations (I gave up mid-way), but even if I did it would not be ideal.

Yes, definitely it won't be a solution for this problem, it should work as previously.

How come this is not the same behavior as on 0.2.0, I thought #46 only fixes the teardown?

This PR fixes teardown (I have introduced regression when removed migrate command usage in Migrator.reset) and improves migration's test setup like described in point 3 in #38 (comment). In #38 (comment) I gave some explanation why it's the best solution from all presented by me in previous comments.

Probably I have used misleading words in PR's title/description and also this issue and solution for it have evolved a lot during all discussions. Firstly I thought it's related only to test teardown and I wanted to keep all this code in Migrator.reset(), but it's not possible, because developer can run tests with many options affecting regular tests teardown/setup, so Migrator must be self-sufficient when it comes to test setup and teardown.

[skarzi]

In current implementation #46 is mainly resolving #40

[skarzi]

@asfaltboy I have digged into Django's migrations deeper and had some answers for your problem, but also many new concerns that I'd like to discuss.

Note that I tried testing #46 locally and I'm having issues with dependency migrations not being applied (while they are being applied in current 0.2.0 ).

This issue is related to how Django built migrations' plan.
In 0.2.0 migration's test setup/assert/teardown flow looks like below:

(NOTE: in both cases, before running tests, django calls migrate command to migrate all forwards).

1. Use migration executor to migrate backward to migrate_from - migrating backward to migrate_from unapplies only migrations that are higher in migrations plan than migrate_from
2. Create some db models instances to test migrate_to on them
3. Migrate forward to migrate_to
4. Do assertions
5. Call migrate command to migrate all forwards

Flow implemented in #46 looks like below:

1. Drop all model's tables
2. Flush django_migrations table
3. Migrate forward to migrate_from - here migrations are applied on clean database and everything goes forward, so only migrations that are dependencies of migrate_from will be applied
4. Create some db models instances to test migrate_to on them
5. Migrate forward to migrate_to
6. Do assertions
7. Call migrate command to migrate all forwards

Let's imagine we have 2 apps: some_app and other_app. some_app has following migrations:

• 0009_do_some_stuff - all migrations from initial to this one are not dependent in anyway on other_app
• 0010_do_some_stuff_dependent_on_other_app - this is the first migration that depends on other_app

other_app is self-sufficient when it comes to migrations (like e.g. django.contrib.sites) and it's much lower in migrations plan than some_app.

Then let's imagine we are testing migration 0010, so migrate_from is 0009 and migrate_to is 0010.
Let's focus on preparation step where some testing db models instances are created.

In 0.2.0 we will have access to models from other_app, because we are migrating backward and only migrations higher in the plan than migrate_before will be unapplied
But in #46 we won't be able to access other_app models, because we are migrating only forward to migrate_from which is not dependent on other_app.

The problem is more trickier than I thought :(
Solution from 0.2.0 is not ideal because it's not similar to how we apply migrations on production, however it's hard to implement production like solution for migration's test flow.

Yesterday I was experimenting with creating migrations plan manually starting from one of:

• plan generated for migrate_to targets
• complete plan generated for clean db

and then truncating such plan on migrate_before migrations, to make it more like on production (avoid migrating backwards) and preserve behaviour from 0.2.0.
It's quite "magical"/complicated solution, however I can clean it and push (but give me few days) so we can discuss it.

[asfaltboy]

Hey @skarzi and @sobolevn thanks a lot for the super thorough and clean implementation in #76 , good work! I didn't get a chance to test it, I'll be sure to try it out on the next migration test we add.

Site node: our "migration tests" seem to be short lived. That is, as time goes by, the dependencies on moving parts grow to become a hassle to maintain. And, when such a migration has been applied in production, (and a few other migrations followed it). So, we end up removing the test, and eventually flatten/remove migrations.

[skarzi]

This side note is so true and unfortunately many people forgot about that.

What do you think about creating some best practices section in our documentation where we can add points like this?

[asfaltboy]

how about something like:

Migration Tests Lifetime

As code ages, so do tests. It's common to see, as the code under test is refactored over time, the test cases become redundant and eventually removed.

This is even more true for migration tests, for two reasons:

1. Once a migration has been deployed to production, and the rollback period is over, the migration code being tested, is never run in this environment again.
2. Migrations often require to keep a copy of the code in model methods, due to that code changing over time. As related components change, this code needs to be constantly maintained (both in the migration and it's tests).

Thus our recommendation is to ...

I'm not sure what's the best practices? Retain the tests only as long as you need them? Continuously clear old migrations (is that a Django best practice)? I guess common sense is to only keep tests around as long as they have a use...