Multi-proxy data migration

[jykae]

• listing the database changes: collections and fields
• design migration(s) to switch to multi-proxy schemas
• dry run & test well locally
  • take DB dump from server that has old release to be able to run several times

Meteor DB backup/restore: http://stackoverflow.com/questions/16816860/meteor-how-can-i-backup-my-mongo-database

Official MongoDB driver API (Node.js): http://mongodb.github.io/node-mongodb-native/2.1/api/Collection.html

Mongration: https://github.com/awapps/mongration

[philippeluickx]

Did we agree on providing a data migration? The other option was to document breaking changes and provide a data migration path AFTER we implemented multi-proxy.

[jykae]

@bajiat @brylie What releases/database versions we need to provide database migration to new release structure? staging, apika, anything else?

[philippeluickx]

We need to really see the trade-off for data migrations for the latest refactor. If it takes less time to recreate the data somehow (or even do some data migrations straight in the DB), then I would suggest to do just that.

But after this release on we should provide data migrations...

[jykae]

@philippeluickx well, that would be relief, and save me from some headache, if documenting the breaking changes would be just enough, and we would provide migration path AFTER these big changes. I could bring this up in daily, what others think.

[philippeluickx]

Yes, please discuss in daily. Also +1 in documenting the breaking changes and maybe also state somewhere clearly that as of (current version) data migrations will always be provided.

[jykae]

@philippeluickx We decided that I investigate first database/schema changes between 0.2.28 -> current develop & give some estimation of the work. We might not need to provide migration path for custom Apika fork, so that makes it a bit more possible to handle this.

[jykae]

@philippeluickx I read about scripting Mongo yesterday & last evening, maybe we could do this bigger change also that way https://blog.kevinchisholm.com/javascript/mongodb/getting-started-with-mongo-shell-scripting-basic-crud-operations/ Maybe that is what you were going to suggest?
It's plain JS that we can feed to mongo shell.

[brylie]

Preferably, the JS code would run automatically when Meteor starts. That way, administrators don't have any manual steps required when upgrading, other than deploying the latest version. That is the strategy percolate:migrations offers -- we automatically run our migration code to the latest version when starting the app.

[brylie]

@shaliko is there a way to hook in to the Docker deployment lifecycle to automatically execute some script(s) in MongoDB shell?

[brylie]

maybe we could do this bigger change also that way https://blog.kevinchisholm.com/javascript/mongodb/getting-started-with-mongo-shell-scripting-basic-crud-operations/

@jykae after reviewing that article, I notice the code looks very similar to what we would write if we used mongodb-migrations. However, using mongodb-migrations gives us a higher level API that is specifically related to migrating database structure, as opposed to hacking together migration scripts.

What do you think would be difficult about using the native MongoDB wrapper in Apinf (which would let us use mongodb-migrations package)?

[jykae]

@brylie Well, it's not that much higher-level imho, it uses official native MongoDB API. Main difference is would we like to do it as migration or not ie. keep track of run scripts & DB version. I could take a look at mongodb-migrations way of doing this, and call migrations to run on startup, if that's the way wanted.

I had some problems with configuring & running that mongodb-migrations previously, when I tried it. Error messages were not that clear what I did wrong. Might have been beginner problems, some small mistake, that's why I proposed percole:migrations back then for simpler usage.
I could try it out again though.

I have written down field mappings as a text file for reference yesterday.

[brylie]

Main difference is would we like to do it as migration or not ie. keep track of run scripts & DB version. I could take a look at mongodb-migrations way of doing this, and call migrations to run on startup, if that's the way wanted.

I had some problems with configuring & running that mongodb-migrations previously
I proposed percole:migrations back then for simpler usage.

Yes, the database code is almost the same. The migration related API would provide a “via ferrata” to safely move from version to version, rather than solo climbing the data migration rock face.

Lets pair program to try out either or both of the migration frameworks under discussion.

Related

• The Strange Brain of the World’s Greatest Solo Climber

[jykae]

@brylie Nice article.
“I’ve always preferred not to look inside the sausage,” he says. “Like, if it works, it works. Why ask questions about it? But now I feel like I’ve sort of stepped past that.” :)

[jykae]

Got mongodb-migrations conf working, and simple run up working with CLI.

Starting next week writing up the migration steps.
Last step will be to add it in startup.

There are open issues with rollback though, so backups before upgrading to next release might be good idea @shaliko
emirotin/mongodb-migrations#13
emirotin/mongodb-migrations#8

[jykae]

I have migration script halfway done with mongodb-migrations. However handling database operations with async.series makes migration script bit more harder to read.

percolate:migrations would be nicer for developer to write migrations.

@brylie @philippeluickx Do we want pure node.js migration code to our codebase?

[philippeluickx]

I am always up for the most simple and elegant solution.

[jykae]

Also meteor mongo driver wrapper hides the real Mongo ObjectID

meteor:PRIMARY> db.proxies.find()
{ "_id" : ObjectId("57ea721f4e8d52093e500b3c"), "name" : "API Umbrella", "description" : "Umbrella settings, migrated.", "type" : "apiUmbrella", "apiUmbrella" : { "url" : "https://staging.apinf.io:3002", "apiKey" : "xxx", "authToken" : "xxx", "elasticsearch" : "http://elasticsearch.docker:9200/" } }

Native driver that mongodb-migrations uses is not doing that.

[jykae]

@philippeluickx ok, I will make tomorrow percolate:migrations solution for the task, and can show also this halfway written migration code if we need to vote.

[jykae]

Sorry I forgot to keep async version with mongodb-migrations for comparison, as I moved the code and refactored to Meteor sync model with percolate:migrations.

I think percolate:migrations is easier to read, and goes better with our codebase. Also easier for every developer to write couple of fields migration for PR.

Anyway async.series migration code looked like this, https://github.com/awapps/mongration#multiple-queries-example
Just with more nested callbacks, and more lines than percolate:migrations version.