Data migration strategy

[philippeluickx]

How do we handle data migrations when moving forward? Do we still make breaking changes? Are migration files needed when changing data models?

Definition of done

• Options for migration have been studies and the optimal solution has been selected
  • Options are documented in the comments for this issue
• There is a Meteor or npm package or migration script that handles database migrations

[bajiat]

Multi-proxy is a prime example of a disruptive, backwards incompatible change on the database level. This is why I am labeling this issue with Multiproxy label.

[jykae]

Option 1: General mongodb-migrations framework, not only for Meteor platform. https://www.npmjs.com/package/mongodb-migrations

Idea is similar like in meteor-migrations package. Plus that this provides CLI tool for running migrations in addition to programmatic approach.

[philippeluickx]

@jykae Is there any reason not to take the meteor standard package?

[jykae]

@philippeluickx well, I understood that Meteor is not providing maintained package. They let community to maintain a solution, https://github.com/percolatestudio/meteor-migrations <-- Checkout the beginning of README. That one is option 2, best of Meteor package options, and encouraged by Meteor Guide. We used that but removed away as it caused problems when updating to 1.3.
I think taking general use migration tool is more likely to stand the test of time. Meteor itself is moving to npm just because Node.js/NPM is larger community.

[jykae]

Basically, they are probably trying the same Django was doing years, before they got lots of feedback about missing migration solution & noticed that they must provide maintained solution for that. With this I mean moving from South to Django migrations in 1.7

[philippeluickx]

@jykae Good arguments, thanks for sharing. I am ok with mongodb-migrations.

[brylie]

@jykae great work! Lets see what we can do with mongodb-migrations.

[bajiat]

@jykae Thanks for the research. I will close this issue and let's discuss concrete steps to take the mongodb-migrations into use,

[jykae]

@bajiat reopening because had troubles get mongodb-migrations working, percolate:migrations might still be best bet for migrations. And selecting package is not yet strategy imho.

[bajiat]

@jykae I'm fine with reopening. Please investigate to come up with the best option for database migration.

[jykae]

I propose we take percolate:migrations back. It's easy to use and works also for Meteor 1.4. Example migration for adding isPublic field if it's missing, https://github.com/apinf/api-umbrella-dashboard/pull/1411

Strategy requires also a mutual decision of process how we handle writing of required migration function when a new feature requires it. It should not be one person fight, who manages the version upgrade into production.

[jykae]

percolate:migrations selected for migrations tool. For upcoming bigger migration separate task for tracing how to migrate data & writing migration(s). Left to decide if it's easier to put it 2-3 migration steps or just one heavier. Amount of data is not large in our existing installations.
As for migration strategy after multi-proxy changes, decided that it's good to take writing migration function as part of PR if it does changes for data model/schema

@bajiat I suppose this can be closed now. I add separate task for multiproxy migration.

[philippeluickx]

We do need to keep an eye on the percolate:migrations package, in terms of maintenance. Currently there is 1 person who proposed to take ownership, but that might change. We also discussed we can always contribute to the package if needed.

[jykae]

@philippeluickx True, I agree, but with knowledge that we currently have I think percolate:migrations is our best option.

[jykae]

Issue #1436 opened for multiproxy migration, to be done, when schema fields are all designed. I assigned myself. Closing this. I think we have all decisions here now.