Lookup migrations tree to find some existing cache

[iurisilvio]

For big projects, migrations are really slow. This project fixes the problem when there is already a cache, but for every new migration it runs everything again.

I want to provide a way to rebuild the database based on "all except these new migrations I pushed now". After that, migrate the new migrations, should be really fast.

To do that, I have to understand how Django construct their migrations tree. I know django-linear-migrations worked with migrations tree recently.

[iurisilvio]

I can't work only with CI caching, because I have to actively lookup if some cache exists. I need an external data store (S3?) to support it.

[iurisilvio]

I refactored the file handling to use django storage to make it easy to just implement a custom storage with django-storages or any other storage lib.

To make it work with my project, I just defined a S3 storage and configured bucket policies.

from storages.backends.s3boto3 import S3Boto3Storage

class MigrateCIStorage(S3Boto3Storage):
    bucket_name = "mybucket"
    region_name = "us-east-2"

[iurisilvio]

I changed the migration hashing algorithm to use migrations tree and expected to be able to just remove last migration to look for older caches, but migrations are not time ordered. I can only trust last migration will run after all their dependencies.

I'll try to find caches without the last migration for each app. It will not hit a cache if the commit has two migrations. If it works, I can try other algorithms for this search.

[iurisilvio]

It works! Version 0.3 running on a big project here, with a S3 storage. Reduced our jobs with migrations from 10 min to a few seconds.

I'm using --depth=1, but don't think performance would be bad increasing to --depth=2.