[8.x] Schema Dump

[taylorotwell]

This PR adds support for a new php artisan schema:dump command which uses mysqldump or pgdump to dump the current state of your schema to a database/schema/{connection}-schema.mysql file.

When this file exists and php artisan migrate or php artisan migrate:fresh is run AND no migrations have run against the database yet (migrations table is empty), this schema file will be loaded into the database first and then any outstanding migrations will be run. This means that effectively this schema file would typically only ever be used during local development or during CI testing. In production, you would typically already have migrations that have run in the past so this schema file would never be triggered.

Once the schema file is created, any or all of your existing migrations my be deleted as long as they have already run against your production environment. They are no longer needed during local development since the schema file exists. For convenience, a --prune flag has been added to the schema:dump command to delete your existing migrations after dumping the schema.

This new feature solves two problems. First, it relieves developers from having a huge migrations directory full of files they no longer need. Second, loading a single schema file is quicker than running hundreds of migrations for each test class in your applications, so your tests can run much faster when using a schema.

[alexbowers]

I haven't had a chance to play with this yet, but i've found mysqldump to have issues with MySQL generated columns in the past, i'll try it out at the weekend if i get the chance, but something worth noting if you haven't come across those problems before

[alexbowers]

Also, worth noting, how does this interact with migrations from packages?

[tobeycodes]

This is really cool. One project some developers in the past prior to joining the project had made changes to production databases manually without migration files and I've been wanting to get migration files in sync with the production database and this seems like it will solve that problem. Thanks!

[kohlerdominik]

We have the same issue as @emielmolenaar . As Seeders are not versioned, it's hard to add data with them in a production environment.

Let's assume we have predefined roles in a role table. Now i released a new Feature, let's say a FlightsList, and i need to add permissions for this to existing Admin-role. So i run this in the migration:

public function up() 
{
    Role::firstWhere('name', 'Admin')
        ->permission()
        ->attach(Permission->firstWhere('name', 'flights.list'));
}

@taylorotwell maybe it would make sense to provide a DataInsertationInterface with abstract public function insert() to it? So we could keep these migrations with $migration instanceOf DataInsertationInterface. php artisan migrate would run all migrations first, and then run migrations with DataInsertation?

[driesvints]

It's always best to extract data changes from your migrations. Your migrations generally don't run twice in your production environment and in your tests you don't want to migrate data every time your migrations are changed.

[emielmolenaar]

@driesvints How can we accomplish that? I think migrations are generally a fine way to add or modify data; however if there any other ways to do this thus making this new feature even more useful I am open to suggestions.

[driesvints]

@emielmolenaar just remove the data modification lines from your migrations when you've run them in production.

[hubertnnn]

How will this affect database compatibility?
Eg. if I want production to run on MySQL but Unit Tests and some of the local envs to use SQLite

[GautierDele]

How will this affect database compatibility?
Eg. if I want production to run on MySQL but Unit Tests and some of the local envs to use SQLite

@hubertnnn You can choose the database connection
php artisan schema:dump --database=mysql

[hubertnnn]

I don't think that will help either if the driver in that connection will change eg.

 'connections' => [
        'main' => [
            'url' => env('DATABASE_URL'),
        ],
        'legacy' => [
            'url' => env('LEGACY_DATABASE_URL'),
        ],
        'test' => [
            'url' => env('UNIT_TEST_DATABASE_URL'),
        ],
]

and following .env:

// On production
DATABASE_URL=mysql://root:password@127.0.0.1/forge?charset=UTF-8
UNIT_TEST_DATABASE_URL=sqlite://localhost/storage/database_unittest.sqlite

// On local
DATABASE_URL=sqlite://localhost/storage/database.sqlite
UNIT_TEST_DATABASE_URL=sqlite://localhost/storage/database_unittest.sqlite

As you can see the connection did not change (is still "main") but the driver did.

My question is if in this case the database dump will be stored in some platform agnostic format and will it work, or will it cause an error.

[alexbowers]

It isn't in an agnostic format, its in the same format as the platform you export in, supporting only mysql and postgres (based on mysqldump or postgresdump being used).

[isclopezm]

I have been doing this "by hand" just to have a better control of the migration files, I never thought about how this may help on tests. Thanks for always be one step ahead!!

[jonathanpmartins]

I just saw this new feature and it really amused me, very nice! It will help a lot of people and save a lot of time. If you have a lot of migrations you can significantly reduce the time to migrate them all.
This otherwise, does not resolve my problem. The number of migrations of the project is pretty close to 1 migration per table. No need to optimize anything here.

My question: - It is possible to decrease even more the migration time for projects with a large number of tables?

I explain It with more details here: https://stackoverflow.com/questions/63855888/faster-laravel-migration-for-projects-with-a-lot-of-tables

[mfn]

Another way I use is:

1. run all tests in transactions (using PgSQL though, not sure about MySQL)
2. only re-run the migrations when they actually change
   A cheap hash function over all the migration files and if that hash changed, I re-run the migrations. Which in practice is only on branch switch.

The 1) guarantees the DB is always in the same state after tests.

Saved me a lot of time when I too had the need to only run singular tests.

[milewski]

I think this feature is pretty nice, however, It seems that won't work at all for those who don't run tests using the same database as the one used in production e.g sqlite:memory vs mysql, as soon as I make the dump a new file called mysql-schema.sql is created ... but that's it.. when running the tests all migrations are gone (pruned) and all tests fail as the mysql-schema.sql won't be loaded as due during tests it uses SQLite .. so I would assume it would be trying to find a file called sqlite-schema.sql instead but as there's one.. it doesn't work...

[driesvints]

@milewski you can choose to dump the file and keep the migrations by not using the --prune flag.

This won't work anymore now that sqlite support has been merged.

[lk77]

Hello,

is it recommended to prune migrations during a major release ? can we consider that users should have run all existing migrations before updating to the new version that remove those migrations ?

they can always go back to the previous version to run the migrations and then update again.

And it will prevent rollback once updated too.

[driesvints]

I personally don't recommend this approach for packages but if you follow semver then yes you'll have to do it in a major release and note about it in a changelog/upgrade guide.

[Bouhnosaure]

Hi, i'm running in the same issue, like @milewski explains, if we prune all migrations we can't setup any other database that are different than the one we use to generate the SQL dump ( and it's logical ) .

However, it is possible to make an export for an sqlite or any other database, but we need to setup this one before and dump it after.

I've tried this and got and sqlite-schema.sql and it's the one that is used when the tests runs migrations ( with a sqlite:memory database )

But what i've got is The command "sqlite3 $LARAVEL_LOAD_DATABASE < $LARAVEL_LOAD_PATH" failed. but in my opinion it's an issue on my side.

So, it could be interesting to dig how we can improve the workflow to dump ( and prune ) when we have a different database for testing.

[driesvints]

@Bouhnosaure since sqlite support has been merged my solution from above won't work anymore. I'll send in a note to the docs to try to explain that this cannot be used in combination with an in memory database.

[Bouhnosaure]

Alright @driesvints, indeed i was a bit sceptical when i've began my upgrade to 8.x for this reason, thanks for your quick answer !

[sdjrppl]

Hi
I'm currently having a problem with migrate:refresh command using previous dump file, mysql .
I'm testing brand new project install. This project already have a php artisan schema:dump --prune file.
When I do php artisan migrate:refresh --seed , first time is ok. Second time It throws error that table A already exists on database. Which means. My schema dump, have migrations previously created, but using --prune , they not exists on migrations folder, like supposed. What I'm missing here ?
Thanks

[sdjrppl]

Well I've edited my .dump file with drop if exists statement before create table. Idk if it's supposed to do this. Using php artisan schema:dump --prune maybe should have this as default behaviour no ? thanks

[emielmolenaar]

Try php artisan migrate:fresh --seed. This will drop all tables first.

…

On Tue, Oct 6, 2020, 17:23 sdjrppl ***@***.***> wrote: Hi I'm currently having a problem with migrate:refresh command using previous dump file, mysql . I'm testing brand new project install. This project already have a *php artisan migrate:dump --purge* file. When I do *php artisan migrate:refresh --seed* , first time is ok. Second time It throws error that table A already exists on database. Which means. My schema dump, have migrations previously created, but using *--purge* , they not exists on migrations folder, like supposed. What I'm missing here ? Thanks — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#32275 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAS3HC3SIFHSV3ROSWYXG23SJMY7NANCNFSM4MDH6FKA> .

[sdjrppl]

@emielmolenaar Dam I've mistaken php artisan migrate:refresh --seed with migrate:fresh --seed . You are right of course , wtf .

[sdjrppl]

nevermind my comment 😕