✨ Migration runner - first iteration

[kirrg001]

refs #7489

This is a pure implementation of the new migration runner.
The connection to Ghost happens in the PR, which comes in a few seconds.

The idea:
Sephiroth (we can rename if we want) has a command line and JS API.
It runs a type (seed, init, migrate) in one single transaction.
It takes over the responsibility to add each migration file into a collection called migrations.

Each type (seed, init, migrate) will work basically the same. It takes all migration scripts in the type folder and executes all.

By storing each script into the database, we can easily check the health of the database.
preTask will check if the migration script was already running.
postTask is triggered when a script finished and stores the script name into the migrations collection.

We still need to ensure that a migration script itself can run twice if it get's executed twice.
So we have two layers of protection. I already took care of for example: owner creation.

isDatabaseOK is a check to ensure seed and init happened. This function is right now not optimised. It just checks if the type init was already executed. Need to optimise that behaviour later when making final changes.

node core/server/data/sephiroth/bin/sephiroth --help

  Usage: sephiroth [options] [command]


  Commands:

    init [options]   populate tables

  Options:

    -h, --help  output usage information

If you remove your sqlite db and run:

node core/server/data/sephiroth/bin/sephiroth init

You can also run it twice, to see that it will skip the init scripts.

This won't work with Ghost yet!Therefor we have the next PR #7502.
So you can just use the runner and then delete your db again.

It uses logging and config, which is a dirty require, but i think that is super OK for now. Everything else will act mostly standalone.

[ErisDS]

Sephiroth (we can rename if we want) has a command line and JS API.

😳
What is one of those?

To clarify here, as I understand:

• init = initial database migrations, calculated from schema.js
• migrate = make an incremental change between versions.
• seed = ? Is this initial, or migration? Do we need one or both?

[ErisDS]

Maybe it's a good idea to create the folder structure you intend for the other parts & use .gitkeep files to keep the structure until we need it?

[kirrg001]

seed = ? Is this initial, or migration? Do we need one or both?

Hmm i didn't really really start with seeds yet, but here is what i thought how it will work:
Init and seed are two different commands. So when having different types (seed, init), it makes sense to require different folders. I want to keep flexibility. Maybe we will have a handful of seed scripts later. I am not 100% sure how much of the fixture logic we will keep.

• init
  • 1-tables
• seed
  • 1-owner
  • 2-welcome post
• 1.0

[kirrg001]

Maybe it's a good idea to create the folder structure you intend for the other parts & use .gitkeep files to keep the structure until we need it?

Hmmm I think it's ok to extend the migration runner in the next iteration for seeds and migrations.

Facts for next iterations:

• sephiroth logic can change in the next iterations to make code easier and re-use more logic of db init logic
• there will be a decision for final folder structure sephiroth expects
• there will be a decision for the old fixture logic
• migrate is straight forward i think

[ErisDS]

Ok - one thing that's worth flagging up here, I think:

The biggest, longest-running pain point that seriously impacted our ability to ship, was being able to safely migrate permissions fixtures. Whilst the code we have for doing that is undeniably large and complex, it did create a very easy way for us to add to the permissions fixtures, which has allowed us to ship more in the last few months where previously we were stuck.

I think it's important to consider that initialising fixtures, and changing or adding to them are two separate things which we will also definitely have use cases for, and that doing it needs to be super simple. Also perhaps worth thinking about the fact that tests also need fixtures (Sometimes we want to mirror the defaults, sometimes you want custom fixtures).

There is a subtle difference here in terminology that I think is making me feel uncomfortable. The concept of "seeds" and "seeding" have been, everywhere I've encountered them (rails, bookshelf), a 1-time thing. Rather than something that can easily be adapted or migrated.

[kirrg001]

There is a subtle difference here in terminology that I think is making me feel uncomfortable. The concept of "seeds" and "seeding" have been, everywhere I've encountered them (rails, bookshelf), a 1-time thing. Rather than something that can easily be adapted or migrated.

Seeds are a one time thing. When your database has reached the "was seeded" state, you can not seed it again. You can try, but it will skip, because each script which was running, is remembered in the migrations collection.

• seeds
  • 1-owner
  • 2-welcome-post

Does this confuses you?This structure was created in my head because seeds are basically just a migration script. And it makes things easier to read when having not everything in one big file like core/server/migrations/seeds.js.

The biggest, longest-running pain point that seriously impacted our ability to ship, was being able to safely migrate permissions fixtures. Whilst the code we have for doing that is undeniably large and complex, it did create a very easy way for us to add to the permissions fixtures, which has allowed us to ship more in the last few months where previously we were stuck.

I think it's important to consider that initialising fixtures, and changing or adding to them are two separate things which we will also definitely have use cases for, and that doing it needs to be super simple.

Initialising fixtures is seeding the database. Changing/adding them is a migration script.

Also perhaps worth thinking about the fact that tests also need fixtures (Sometimes we want to mirror the defaults, sometimes you want custom fixtures).

Ja sure!

I am a bit confused about your comment, maybe we can talk in DM shortly, not sure what you are going to point out here 👍

[ErisDS]

👍 let's chat, nothing serious, just historic info / knowledge to share.