Method for updating data on version upgrade

[aaronjudd]

Add some kind of trigger when a package version changes to allow programmatic migration of existing data to any new updates.

ie: Update from v0.2.1 -> v0.2.2
Trigger: Move shipping object out of Settings collection and adapt, insert into Shipping collection

[bluerabbbit]

+1

[aaronjudd]

@jshimko we should review this requirement as part of the plugin/module architecture.

[aaronjudd]

@zenweasel I believe this is the issue where you would want to discuss migration implementation details. There's also some background discussions worth reading in the import issue #467

[zenweasel]

I have come into a scenario where I feel like I need to update data on version upgrade. Looking back it's sort of needless but oh well.

In the last version we built versions of the OrderSearch and AccountSearch index collections even though they weren't used for anything. But now in this version with actual Order and Account search the collections are different and the search will error out if the old collections are in place. (tested and verified)

So I added, just as something to look at a very simple implementation of migrations using the percolate:migrations package. You can see it here https://github.com/reactioncommerce/reaction/tree/f0e037b550d651f4826c188632383a35f3a2441f/imports/plugins/included/migrations

I'm fine if we want to throw this away, just wanted to try something and see how it actually worked in practice.

[aaronjudd]

I know that this probably requires a more in depth discussion, but I also think we don't need to over engineer this, as long as we leave ourselves some easy upgrade opportunities.

I'm sure there's far more robust solutions, but this gets the job done immediately in #1494. Not sure if this is correct, but I tend to associate migrations with data, and versions with update or upgrades of software, so I also prefer your "versions" naming better.

Looks like we can configure collectionName: "migrations" as collectionName: "versions" to mirror your convention of versions elsewhere. If you renamed the folder "migrations", "updates" could be used instead of "versions" for the actual imports and that would make sense.

I'd suggest that it'd be ok to move from included to core, I can't see us wanting this to be optional.

import { Migrations as Versions} could default export, so that you can just include and export

import { Versions } from "imports/plugins/core/versions"

I think that could make it a lot easier to reuse in other plugins without things breaking if we replace (and that's pretty likely). I'm assuming that you can have multiple instances of Migrations.add?

Can this support our versioning from the package.json? (ie, 0.17.0)? That way we'd be using a consistent reference.

Maybe we could use the version direct from reaction -v, ie: have the reaction-cli export checkVersions() whenever it's run so that we could then use as global application version. (and maybe update an entry in the registry, for other use cases). I'm not sure how we'd handle that for production builds, meteor cli.. or if we're exporting this somewhere/somehow that I've forgotten.

The actual version of the app's running package.json is 0.0.0 because it's a fake package that the Meteor build creates, so the env export stuff is a bit of a work around, but it'd also be useful functionality as we start moving plugins to npm packages in the future, as we'll be able to use the version from their manifests in the registry as well.

What happens on failure? Is it graceful?

I'd be more comfortable if this was adapted to npm (as in their "Call for Maintainers" issue), as I think architecturally we should be moving as quickly as possible away from any Meteor packages. Maybe something we need to shepherd if need be?

[zenweasel]

One issue that I thought of is that when I do a PR that may have a migration in it I may not know which "version" that code may be in. Let's say it's a bug fix and we may issue a point release or we might wait until the next release depending on how severe, etc.

This is probably a process issue rather than a technical one but it would be one argument against not tethering the migration version to the software version (at least not in the way that I have it here)

Maybe it's just an issue where the release lead just needs to cleanup/align all the migrations with the release to prepare it for merging.

[zenweasel]

To answer your question about errors, I tested this this morning (totally on purpose of course). If a migration throws an exception that migration halts and that migration record is marked "locked" and migrations will not move forward until that migration is unlocked (which is done by just manually editing the record). So depending on where it failed you could mark that migration unlocked (so it runs again) or completed.

[aaronjudd]

@zenweasel to capture our chat, the plan is changed a bit.. to use migrations as the namespacing. I think we then also should use, Migrations (capitalized, as the collection name). Since the migration packages isn't really version based, we'll probably want to relate this to versioning in the future, but the migration.version itself should the "version of the migration", not the application package version. Thus, ordered integers starting with 0 (un-migrated), 1 , 2, 3 ,etc.

[zenweasel]

@aaronjudd Should we close this issue and just deal with specific issues/enhancements as they come up.

[zenweasel]

Closed via #1494. Simple version using percolate:migrations