Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Split a single migration into multiple files #615

Open
0xjac opened this issue Aug 20, 2021 · 0 comments
Open

Split a single migration into multiple files #615

0xjac opened this issue Aug 20, 2021 · 0 comments

Comments

@0xjac
Copy link

0xjac commented Aug 20, 2021

Is your feature request related to a problem? Please describe.
Some of our migration files are very large. (E.g. The initial up migration file, defining most of the schema, significant refactors of the data model, collapsing of migrations, etc.)

Describe the solution you'd like
The possibility to split a single migration into multiple ordered files (whilst keeping all files as a single migration version).
Essentially this would require modifying the file name format from:

{version}_{title}.up.{extension}
{version}_{title}.down.{extension}

to:

{version}_{part}_{title}.up.{extension}
{version}_{part}_{title}.down.{extension}

Where part is an optional incrementing integer for its migration version. This would result in something like:

0001_01_users.up.sql
0001_01_users.down.sql
0001_02_base.up.sql
0001_02_views.down.sql
0002_triggers.up.sql
0002_triggers.down.sql

From a clean database, running all up migration would run the files in the following orders:

  1. 0001_01_users.up.sql
  2. 0001_02_views.up.sql
  3. 0002_triggers.up.sql

Note that:

  • It should not be possible to run some of the files for a migration versions. All of them must be run together. (Otherwise, it would just be separate migrations...)
  • Splitting the code across file obviously implies separate transactions (when applicable).
  • If a version specifies parts for up or down, then all files for that migration must specify part in order to have a deterministic order for the files. (I.e. you cannot have 0001_01_users.up.sql and 0001_views.up.sql.)
  • Part numbers can be skipped in one direction with respect to the other direction, similar to migration versions.
  • By keeping file parts optional, existing migration files will continue to work as expected, thus maintaining backwards-compatibility.

Describe alternatives you've considered

  1. Simply split the large migration into multiple migrations.
    This does not work for me, as there is a risk to leave the DB in an inconsistent state due to user error, as it is unclear which migrations should always be applied in a group.
  2. Use an external tool to generate a single migration file from multiple files (either by simply concatenating multiple files or using templates).
    This does not work either. Besides the overhead of build (and remembering to build) the migration files, this introduces a lot of complexity and undermines the consistency of the migration files.

Additional context
Splitting large migrations into smaller files might mitigate #300.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant