The new roadmap to v1.0.0

[wzrdtales]

Foreword

This year is special to me, since I am finally able to provide more of my time to open source. First of all due to the commitment #605 of sponsoring time and money from my own company (which currently means effectively plan in whole days/weeks of time that I do not spend on paid projects and invest those into open source instead). This is important, since nothing is free in this world and this is my new way, to give something back to the open source community, since a lot changed for me and I do not have as much free time as I wish I would have.

Further I added opencollective and sponsoring slots, from which funds also solely progress and work on db-migrate will be paid. So this way also people that are not able to contribute to the project in terms of providing PRs and their workpower, can contribute to the success and further development by providing resources otherwise.

That being said, i am very grateful for all the contributors of the past and of the future of any kind. All support matters!

The roadmap to v1.0.0

That being said, the new roadmap which we try to achieve over the next few month consists out of the following:

• Adding a concurrency manager
• Adding a state manager
• Adding internal schema representation
• Make db-migrate learn about the schema
• Introduce the new v2 migration schema
• Rework db-migrates structure and components
• Deprecate node <8
• Add chainable steps for new migration schemas
• Add support for multiple driver instances
• Add new methods to the drivers for the state table creation (this time without any future need to adjust again)
• Make sql migrations first class, but separate them into plugins (no more javascript loaders, wont be in the standard features anymore, and will be needed to installed once as a plugin)
• Adding new templates and move them to plugins
• Clearly define the new concept for seeders (in work, but will integrate more seamlessly than the current approach)
• Adding default columns to table creation, such as an indexed seeder key column to reference and control seeded content and as well support easier support for reproducible system in integration testing
• Add separate static seeds for very simple needs, without any version control (still making use of the default columns)
• Enhance the documentation in general
• Add the new features to the documentation
• Remove transitioner from the project

Work In Progress

The good news here are, a lot of this is done or in progress (master branch, not released yet). For example the structure has been reworked completely already. Yet I am thinking to also replace the CLI, but in favor of a shorter release time I might drop that into the next step. What is in its current state already ready is the new migration schema with a lot of new features. More to that later. The next step will be the templates and the default columns for table creation. After that I either go on the seeders or the concurrency manager, which highly depends on how the dependencies of this task resolve to each other (it turns out, most if not all tasks are somehow related to each other and require each other).

Also this is a very intensive rework and a lot of work put into thoughts and new processes, I am looking to keep the breaking changes to a minimum. In fact, I will never drop the version 1 migration schema. So expect that most should continue working, except that we're completely dropping support for old node versions, to move the db-migrate code base to a more modern state. This should be fine for the majority, for everyone else that does not integrate the db-migrate api, this change shouldn't affect to badly either. But it is finally time to move on and drop the support at this stage (also noting that node v6 is almost out of the maintenance cycle anyways.

The new migration schema

So the biggest thing in this release will be that I will finally clean up with the missing seeders and get the strings straight on this point and of course the migration schema. Which is leveling up the game a lot.

So to give you a preview of what is coming (and already working, if you're feeling fancy checkout the master branch and pg driver which already carry support). Here is what it is about:

The migration schema v2

The migration schema v2 is a very strict new standard, in which ultimately only reproducible components are allowed to be executed. It adds support for atomicity without transactions, it is to a certain degree self healing and it massively decreases the complexity of creating migrations. It allows an as seamless process of migrating up, as rolling back. It clearly defines guidelines for zero downtime compatible migrations and some more compelling features which will incorporate the DDL as well as the DML.

To give you a more alive example, lets take a series of actually working v2 migrations.

So assume this two migrations:

Migration 1

'use strict';

exports.migrate = async (db, opt) => {
  const type = opt.dbm.dataType;
  await db.createTable('test', {
    id: {
      type: 'UUID',
      primaryKey: true
    },
    test: {
      type: type.INTEGER
    },
    test2: {
      type: type.INTEGER,
      notNull: true
    }
  });
  return db.createTable('test2', {
    id: {
      type: 'UUID'
    },
    test_id: {
      type: 'UUID',
      foreignKey: {
        name: 'namename',
        table: 'test',
        mapping: 'id'
      }
    },
    test: {
      type: type.INTEGER
    }
  });
};

exports._meta = {
  version: 2
};

Migration 2

'use strict';

exports.migrate = async db => {
  await db.dropTable('test2');
  await db.removeColumn('test', 'test2');
};

exports._meta = {
  version: 2
};

Explanation

So what you can see here is, migration 1 creates two tables and migration 2 deletes one of the table completely and and removes a column from the other one. The first thing you will notice is that the interface looks different. First of all there is no up and down anymore. Only a migrate and a required _meta export are provided. The latter will later also allow to introduce some more features like specific driver instances per migration later on. The second change in the interface is, that callbacks are dead. Instead the migrate function now gets passed the driver and an options object. The options object contains the objects you used to retrieve through the setup function and might also hold some other toolings in the future.

So all in all the migrations look and feel very simple now and you might ask, where is my definition of rolling back? The answer is, there is none and you don't need any. The reason for this is the same reason why the new standard is so strict and the same reason why the v1 standard will keep alive forever. Certainly everything can be covered in the v2 schema, but in the meantime it will always be possible to execute operations that are not yet executable on the v2 schema. But that aside. Why you don't need the down operation anymore?

db-migrate got a few new components exclusively available to the v2 schema (with some exceptions such as manual schema feeding which will be available for inter compatibility to the v1 as well). These components are for example a component that learns about the manipulation of the DDL you're applying currently and another that stores the state of the new manipulated schema and information about the alteration that has been made. When you now execute an action in the new schema you're not accessing the driver directly anymore. Instead db-migrate will inject a chain of actions before it sends your operation finally off to the driver beneath. This allows db-migrate not only to intercept but also to correct actions. For example the new migration v2 schema also comes with failureStrategies. Currently the only available and also the defacto standard is rollback the current migration on failure automatically. Should the migration being interrupted by for example a failure of the database in the process of migrating, then db-migrate will be able to recognize this on the next run and rollback the executed actions before continuing by default.
The rollback is being recreated from the stored information. That also means, when you decide to delete a table for example, db-migrate knows exactly how your table looked before it was deleted and can completely automatically revert to the before state. An exception to this, is of course the data in the table. This is why a dropTable option is not safe by the zero downtime guidelines and a new deprecateX function for tables and columns will be added to the driver. Those will support a full zero downtime compatible cycle. Which will consist out of the following step:

• 1. mark the entity to be deleted on the first round.
     Now you will make your first release in which you will implement support for the new schema coming up.
• 2. rename the entity with an unique name
     Now you will release the new version which uses the new schema instead and your old version already respects this in which way ever necessary.
• 3. after a cycle of by default 4 releases (including the 2 before already) db-migrate will remove the table

These deprecation functions will also get later on support for signaling from the application directly, to shorten the process and they require information about a new release. For this there will be _meta export signal to define a certain migration as a start for a new release. This also might be improved in the future, since there are plans to also implement new migration patterns, to organize migrations in features and versions directly (for example a long requested feature like semver versioning might become reality, but this is yet to be defined).

Some final words about the standard itself. There will be a definition of the standard, which might be extended with features, as long as it is not altered. So already defined components might only change in a new version of the schema, which shall be avoided when possible. But new components might be added to make it more feature rich.

Some more notes

As always I want to express, I welcome any help. Be it documentation, implementation or helping out others with their issues. I want to thank all the people that helped in the process and all the contributors that added value to the project and I am looking forward to make db-migrate a migration framework that is not staying in your way, but helping you to do the right things and get the things done. Especially important to me is to search for a way to transfer knowledge, especially on the zero downtime section. I don't know yet, how I best accomplish this, but I consider this being even more important than everything else since I might provide a powerful tool, but what does it help you if you don't know how to handle it. Whoever has ideas or wants to help out on that section. I greatly appreciate your input!

Post v1.0.0

So to answer the question, what comes post v1.0.0. I can't answer yet, but there will always be room for improvement and i want to encourage you the community, to take active part in creating meaningful value for the project.

Notable references

• RFC #5 - New Migration Schema #508
• RFC #1 - questions about supporting test database syncing better #324
• RFC #3 - Should be able to just have SQL files #401
• Post 0.10.0 Release - Tasks and some statement #518
• State Manager #538
• Retarget the shadow driver #539
• Rewrite loader #537
• Move templates to plugins #514
• Question: concurrency #464
• Support "databaseless" execution commands #338
• Implement State discovery #233
• Version controlled seeder #215
• Integrate Migrations with Seeders and vice versa #230

Issues to be checked further

• _getKV for modSchema should not overwrite default when empty #664

---

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

[wzrdtales]

/cc @BorntraegerMarc

[wzrdtales]

To add some afternotes, v1 migrations will also benefit from concurrency control, should the driver beneath support it, but they won't be able to automatically cooperate with v2 schemas. For that manual feeding of the schema will be added, so that it is still possible, to run rawSql and apply the alterations made manually.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[wzrdtales]

some significant progress lately.

• templates are there now instead of any templating language, we will just let the plugins decide how to render the template. a hook for cli options will be added as well. Internal templates will just use es6 templating and we move each template to its dedicated file.
• v2 schema is maturing and a lot of bugs are fixed, it is being used in production right now in some of my biggest projects and clients. still in dog feeding phase, but the worst bugs have been identified and fixed by now
• state manager is fully working by now, just missing the concurrency manager

next up is

• concurrency manager
• seeder concept (and maybe implementation already)

[BorntraegerMarc]

@wzrdtales would it make sense to release a release candidate? Of course only if you think API is not gonna change :) I would be glad to test a RC in my biggest projects as well.

[wzrdtales]

there is a beta tag released already :)

[BorntraegerMarc]

there is a beta tag released already :)

yeah, saw that :) I was specifically referring to a release candidate because I would like to wait with adoption until we're sure that the API is not gonna change anymore. I can invest some time to test pre-releases but don't have time unfortunately to re-implement changes, when they occur...

[wzrdtales]

I will make sure to hit you up as a test candidate as soon as I freeze the standard (actually lot of it is alreay freezed).

which drivers are you using btw.? currently I implemented the support for the new functions only on pg and cockroachdb.

[BorntraegerMarc]

which drivers are you using btw.?

I'm using MongoDB

[wzrdtales]

Also decided to remove the transitioner entirely. It was a utility to help migrate away from the very old async libs and globals from the first incubation before I rerolled the package completely. Will be available in the older versions should anyone need it to still migrate away pre 0.9.x migrations.

Note to myself: add to breaking change list.

[wzrdtales]

tunnel ssh will move into the plugins, again

Note to myself: add to breaking change list.

[dimitriy-tomilovskiy]

@wzrdtales Any news on concurrency manager? It could be mighty useful for some of the things we are looking to do.

[levino]

Where are the docs for the beta versions? I was unable to find them, sorry...

[rysolv-bot]

themanmaran has contributed $75.00 to this issue on Rysolv.

The total bounty is now $75.00. Solve this issue on Rysolv to earn this bounty.

[nilligan]

Would it be possible to set a different, non-latest tag to the beta releases? The current latest tagged release is for 1.0.0-beta.14, which means that anyone running npm install db-migrate installs a version of the library that is still in beta (rather than latest stable 0.11.11)

From the npm-dist-tag docs: https://docs.npmjs.com/cli/v6/commands/npm-dist-tag

Publishing a package sets the latest tag to the published version unless the --tag option is used. For example, npm publish --tag=beta.
By default, npm install (without any @ or @ specifier) installs the latest tag.

It also produces (what I assume is) unwanted behavior from the npm outdated command

> npm outdated
Package                    Current   Wanted         Latest  Location
...
db-migrate                 0.11.11  0.11.11  1.0.0-beta.14  <redacted>
...

[wzrdtales]

ahh i hate npm for being inconsistent... It was indeed not wanted, I already wondered why suddenly more people are starting to use the beta. A tag created with the prerelease flag really should just being treated as such without the stupid --tag flag... (it was previously published as thus, but I didn't use that flag a long time and just forgot about its existence again assuming npm was designed intelligently)

I will see to push a "new" old version out, however, the latest version is pretty stable, it is in use in huge production systems (however many of them maintained by me where I am dogfeeding the new migration schemas and by that battle testing them before the release) and there are really not much issues, especially if we leave out the new functionality.

[nilligan]

Totally agree on wishing that npm treated the latest tag differently...

Would be very much appreciated if you could publish a 0.11.12 and add a different --tag option to future beta releases

And thank you 🙏 for all your work on testing v1.0.0!

[mriedem]

Hi, I'm just wondering about any kind of status updates on the 1.0.0 effort. We're probably going to upgrade from 0.11.12 to the latest beta (currently 1.0.0-beta.18) because we're getting vulnerability scan hits on old transitive dependencies. The beta releases have been happening for 3 years now and the last beta release was 5 months ago. I'm assuming the beta versions are pretty stable (given they were considered stable in 2020 per the common above), I'm just wondering if there is an expectation of when the beta tag is dropped and 1.0.0 becomes official?

[wzrdtales]

v1 is for long time in use in our projects where we battle test it, there are still some issues with the v2 migrations although it is very stable by now, if you rely on the v1 schema but use v1-beta of db-migrate it will be rock solid though. We have been scaling for the last period a lot company wise, so that ate a lot of time, we still are involved in db-migrate and continue to support it. It might be really time to consider dropping out v1 without the final support of the v2 schema, I would need to check if there was any breaking change pending to avoid another major version jump.

[wzrdtales]

Also worth mentioning, we are the biggest investors as a company behind db-migrate right now, but we support more than just db-migrate as a open source project, so it is not like we could expand the efforts we already make (also not everything is visible directly to everyone, especially considering the v2 efforts and driver related things like the stuff on the cockroachdb driver). So obviously it would help the project to get more backers and funding, and individual contributors to accelerate the pace.