Plan & test MongoDB database migration from 2.x to 3.x

[jykae]

Meteor upgrade brought upgrade to MongoDB for new database engine. See MongoDB documentation for migration steps.

"Your development database is using mmapv1, the old, pre-MongoDB 3.0 database engine. You should consider upgrading to Wired Tiger, the new engine. The easiest way to do so in
development is to run meteor reset. If you'd like to migrate your database, please consult https://docs.mongodb.org/v3.0/release-notes/3.0-upgrade/"

In DEVELOPMENT, use meteor reset to switch to WiredTiger based

In PRODUCTION, follow the procedure by MongoDB docs.

GENERAL PROCEDURE https://docs.mongodb.com/v3.0/release-notes/3.0-upgrade/#general-action-procedure

• take database dump from old
• upgrade mongo
• restore data to upgraded mongo that uses WiredTiger

IMPORTANT:
"To change storage engine to WiredTiger, you will need to manually export and upload the data using mongodump and mongorestore"
https://docs.mongodb.com/v3.0/release-notes/3.0-upgrade/#change-storage-engine-to-wiredtiger
=> Old DB engine cannot read Wired Tiger data format.

Test procedure on staging/test server first.

[jykae]

Mongodump & restore with Docker, https://blog.studiointeract.com/mongodump-and-mongorestore-for-mongodb-in-a-docker-container-8ad0eb747c62#.vtvgxojzb

[jykae]

Successful procedure on my local virtualbox production environment.
Next step, do the same on nightly.apinf.io

[jykae]

Verify engine in mongo shell:

db.serverStatus().storageEngine

[jykae]

nightly.apinf.io upgraded to 3.2.6 and using wiredTiger

[jykae]

Reopened for need to test also deployments with new docker-compose based setup

[bajiat]

@jykae Should I co-assign @shaliko to this task?

[shaliko]

@bajiat @jykae I already tested migrate on version 3.2.9, but not tested migrate on wiredTiger engine https://github.com/apinf/api-umbrella-dashboard/compare/update-mongo-version?expand=1

[shaliko]

I suspect for migrate existing data on wiredTiger, we need do dump data and after upgrade import that dump - need free time for check it.

[jykae]

@shaliko yep, right. Meteor bundles MongoDB 3.2.6 for development, would it be good to keep it same as on dev?
I noticed on upgrade that MongoDB is smart enough to check from files what engine is used and without removing database folder completely, it continues using mmapv1 engine. I could share shell scripts for backup & restore for you, that needed some research how to do that when Mongo is inside Docker container.

[jykae]

As discussed on the daily, this task can be considered done on my side. @bajiat said that @shaliko will have task for upgrading MongoDB on nightly2, backup/restore scripts shared. I am available for consultancy though if any questions.

[shaliko]

@jykae I used 3.2.9 because official docker images dose not have 3.2.6 https://hub.docker.com/r/library/mongo/tags/

[bajiat]

The related task for Docker assigned to @shaliko is #1410.

[jykae]

@shaliko ok, that's patch level version so that might be ok. I wonder how I then was able to pull with tag 3.2.6 for nightly ?

8910f56dc474        mongo:3.2.6                      "/entrypoint.sh mongo"   4 days ago          Up 4 days           127.0.0.1:27017->27017/tcp      mongodb

[jykae]

I am closing this.

[shaliko]

That is funny, missed on Docker hub, but I reproduced that we can download that image

docker pull mongo:3.2.6
3.2.6: Pulling from library/mongo
47994b92ab73: Downloading [============================================>      ] 32.78 MB/37.2 MB
...