Store state in the DB, not in a file.

[gtmtech]

Storing the state of which migrations have been run in the DB itself, rather than in a file may be preferable in a number of circumstances:

• Running large clusters
• Running phoenix servers

The trouble with a local file, is you pin your migrations to that machine (or you need to copy the file around). This may not always be possible in some setups. It would be good (as in rails activerecord migrations framework) to be able to store the DB state in the database itself, perhaps via a flag to the migrate command or something like that.

[chrisspiegl]

I think this could be done via n option. But it is not that easy since node-migrate is cross database. So everybody would have to implement the process him/her self.

[arnorhs]

another benefit to doing it the rails way is that if your database gets corrupt, or you have to restore from backup for some reason, your list of run migrations and the current db state won't go out of sync.

[SystemParadox]

+1

[cizar]

+1

[joelnn]

+1

[joshperry]

@arnorhs But the whole point of storing the migration point with the data is that you know what migrations have been applied to that data once you've restored from backup. If you don't store the migration version with the data then how do you know which migrations have been run on a specific restore?

[joshperry]

I would be willing to put some time into componentizing the migration state persistence, if there's a chance of the PR being merged. I just don't have the time/inclination to manage a perpetual fork.

The user already obviously has a connection to the data store so that they can run their up/down migrations. It would be pretty easy to allow the user to provide a function to persist an opaque migration state, and one to provide the current migration state back to the system when the migrations are run.

[joaosa]

@joshperry If you're willing to work on this change, I could merge it for you.

[toymachiner62]

👍

[gobengo]

@joaosa @joshperry @gtmtech What do yall think of #42 I just opened, which enables migration mixins that can, for example, store migration state in a DB?

[toymachiner62]

I think it's a good workaround, but IMO if i have to remember to specify a flag to store my migrations in the database rather than a file, I feel like this is very error prone.

[joelnn]

Just a note, it's important that the migration state be saved in the same database that the migrations were applied to, e.g. saving mysql migration state to mongo would not be a good idea. You can have a single application that has migrations for different databases.

Also for databases that support transactional ddl changes, the migration state probably should be committed in the same transaction as the migration.

[santthosh]

+1

[Jokero]

+1

[grushikhin]

+1

[LinusU]

The discussion in #42 might be of interest. As I said there I will try and do a writeup on storing state as soon as I get the time, hopefully this can be fixed in the not too distant future. Thanks for the patience everyone!

[gustavnikolaj]

I'm not sure if this has been suggested, but what about just allowing migrate.load to take the state as the first argument, instead of only accepting a path to a file.

Instead of:

var migrate = require('migrate');
var set = migrate.load('migration/.migrate', 'migration');

Just doing:

var databaseState = someMethodThatQueriesTheDatabaseForTheMigrationState();
var migrate = require('migrate');
var set = migrate.load(databaseState, 'migration');

That would require people to write their own wrappers, when they want to do something that is more complicated than the general case. But it would on the other hand, also allow the library to stay really simple, and very general.

Just a heads up: I have not yet used this library, but I'm about to start a project where I will need a migration framework, for a database that do not currently have any good tools for that problem in node.js.

Edit: I just realized that you need to read and write the data of course. Reading is not enough. But you could pass in an object with a read and a write method instead of the path. That would have the exact same pros and cons.

[gustavnikolaj]

A proof of concept of the above idea: master...gustavnikolaj:feature/stateAsInput

There's a little thing that I didn't take care of, and that's the serialization in the save method. Originally this code just does a JSON.stringify(this). But apparently, it only ever uses the Set.pos variable, so we could choose only to persist that.

All tests are passing, and it is completely backwards compatible.

[marvinroger]

Another advantage of storing data into the DB is that it would be compatible with PaaS without persistent FS storage or even Docker containers. This issue definitely needs to be solved!

[bestan]

+1

Changes @gustavnikolaj proposes look good. Do you think it can be merged?

[gustavnikolaj]

Earlier in the thread, @joaosa proposed that he could merge a similar PR if @joshperry would do it. I'll happily send a PR of my proposal, if that offer still stands.

My fix is completely backwards compatible, but allows people to supply an alternative implementation than the file based one that is shipping with the module. Database specific implementations can be shipped as standalone modules or written on an ad hoc basis.

[jimmed]

+1 to @gustavnikolaj 's proposed changes. Please raise a PR, as this would be extremely useful to me!

[gustavnikolaj]

@jimmed and others that might want to use this prior to my PR referenced above being handled: It's released on NPM as @gustavnikolaj/migrate if you want to try it out.

[jimmed]

@gustavnikolaj Thanks, that's absolutely perfect! So far, so good.

[wmartins]

+1 for @gustavnikolaj pull request, it would be useful to me and to my team

[lauort]

• 1 for @gustavnikolaj pull request, it is perfect to check the update states.

[okv]

Hi, there.

I maintain east - migration tool with main ideas to store executed migration names inside db and provide existing connection to the db for using it inside migration. It has adapters for mongodb, postgres, mysql, sqlite. I and my colleagues using it a lot at our day to day job. It could be useful for someone else. For me such missing feature in node-migrate was (couple of years ago) one of motivations to make another tool =)

[darkbasic]

+1

[blueror815]

Hi.
I want to set mongodb config in migration file.
How can I do it?

[arturskeeled]

This is perfect for my needs, but since I use AWS CodeDeploy together with CodeShip for deploys, it doesn't work unless persistence is added to the DB, not to a file.
Is there any update on this?@tj
@gustavnikolaj I downloaded your codes so i could do the save/load on my own, but I can't make it work for some reason.

If any of you could help, it'd be much appreciated. Tnks

[jimmed]

@arturskeeled I had the same needs - we were deploying to Heroku and so needed to persist database version in the database. As it was for a former employer with some more bespoke needs, I ended up writing my own little migration tool based on node-migrate, but I wasn't able to open source it before the company folded. Sorry!

[arturskeeled]

@jimmed I'll start doing the same right now. Thank you

[lauort]

I can help If you want
Am 13.06.2016 09:46 schrieb "Artur Teixeira" notifications@github.com:

@jimmed https://github.com/jimmed I'll start doing the same right now.
Thank you

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#29 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AHCPrp_iwIFWVyJh6lI1KMIb9lGHYlvUks5qLQrDgaJpZM4BM9HK
.

[arturskeeled]

It was actually super straightforward since i wanted a really simple example.
I can leave the pseudo-code here.

1. Get all files from certain path.
2. For each of them search the DB to see if id/code/name is already there.

3.1 If it is, cool, do nothing.
3.2 If it isn't, require the file and run the up, if that failed run the down.

3.2.1 If everything went okay, save the new id/code/name in the DB in a table called migrations or whatever you want to call it.

This was my logic, hope this helps.

[antony]

We'd also love this to work now. Is there a chance that the code could be ported from @gustavnikolaj 's version to the current codebase? Happy to help if need be...

[wesleytodd]

Ok, there is an implementation of this in the 1.x branch #77. If you could all check that out and give feedback that would be great. thanks!

[lewisl9029]

Per @joelnn's comment above:

Also for databases that support transactional ddl changes, the migration state probably should be committed in the same transaction as the migration.

Can the current implementation in #77 support this use case? Is it as simple as starting the transaction at the beginning of load, committing the transaction at the end of save, and then rolling back in the error handler? I just wanted to make sure I'm not missing some nuance in the timing of when these functions are called. Thanks!

[wesleytodd]

I have not fully thought through this, nor tried to make a proof of concept, but I think you would need to setup a wrapper yourself which integrates to your state store. I think you would actually want to start the transaction at the migration event and commit/rollback on save.

The migration scripts do not have access to the set, so you don't directly have the events you need. You would probably have to implement a custom runner with the current code.

If someone would like to build an example store which implements this I would be happy to discuss ways to make it more directly supported, like maybe giving the migration access to the set.

[wesleytodd]

Hey, so this is fixed, and there is even an example of doing it in the repo. Going to close this.