First pass at migrations #64

sgrif · 2015-12-20T18:21:20Z

This is still missing docs, and I want to flesh the feature out before we ship it, but this is a pretty reasonable sized chunk that I think is ready to be merged on its own.

This adds the ability to manage the database schema by running migrations. Migrations exist in a /migrations directory. Each migration is a timestamped sub directory (like 20151220_create_stuff) containing up.sql and down.sql.

When a migration is run, up.sql will be executed. There is currently no way to revert a migration, though most of the infrastructure for it is in place, and it will run down.sql. An internal table is maintained to ensure that migrations are only run a single time.

Migrations can be run using the run_pending_migrations function, which is meant to be run from a build script. One downside to this is that any console output will get eaten unless the thread panics. I eventually intend to add some command line utilities for these tasks, including generating new migrations.

The migrations themselves have no specific integration tests, other than the fact that they are used by our integration test suite to set up the schema. This gives a good smoke test on CI, but unfortunately means that the database needs to be dropped locally if this code changes, to ensure the migrations are re-run. We should eventually improve this.

sgrif · 2015-12-20T18:22:33Z

Review? @mfpiccolo @samphippen @mcasper @mikeastock

fables-tales · 2015-12-20T18:25:26Z

diesel/src/migrations/migration.rs

+use std::path::{Path, PathBuf};
+
+pub trait Migration {
+    fn version(&self) -> i64;


Could this be a string? Feels like the property you want here is sortable, not "only a number". For example: leading zeroes might be useful, or users might want to have hexadecimal or similar. Thoughts?

I have one concern with that. If we assume it's always a number, we have a clearly defined way to determine what is and isn't a migration in this directory (for example, we shouldn't error when we see .gitkeep). We'd need to make sure we have a strict definition, as I do want to error in the case where something looks an awful lot like a migration, but is off somehow (down.sql being missing for example). Is ignoring files with a leading . enough here?

yes on ignoring .

mfpiccolo · 2015-12-20T23:11:09Z

Having a little bit of trouble getting set up on my test project. I got the migrations to run from build.rs, but I am now getting:

src/main.rs:1:1: 1:1 error: use of undeclared type name `users::email` [E0412]
src/main.rs:1 #![feature(plugin, custom_derive, custom_attribute)]

when using infer_table_from_schema!(dotenv!("DATABASE_URL"), "users");

Or:

diesel macros>:20:19: 20:33 error: use of undeclared type name `columns::id` [E0412]
<diesel macros>:20 type PrimaryKey = columns:: $ pk ; type AllColumns = ( $ ( $ column_name ) , +
                                     ^~~~~~~~~~~~~~
<diesel macros>:15:1: 17:58 note: in this expansion of table_body! (defined in <diesel macros>)
<diesel macros>:5:1: 6:2 note: in this expansion of table! (defined in <diesel macros>)
<diesel macros>:2:1: 2:64 note: in this expansion of table! (defined in <diesel macros>)
src/models/mod.rs:1:1: 1:40 note: in this expansion of table! (defined in <diesel macros>)
<diesel macros>:20:19: 20:33 help: run `rustc --explain E0412` to see a detailed explanation
<diesel macros>:22:33: 22:47 error: unresolved name `columns::id` [E0425]
<diesel macros>:22 & self ) -> Self:: PrimaryKey { columns:: $ pk } fn all_columns (  ) -> Self::

when using infer_schema!(dotenv!("DATABASE_URL"));

sgrif · 2015-12-20T23:23:13Z

@mfpiccolo Do you have a primary key other than id on that table?

mfpiccolo · 2015-12-20T23:38:12Z

Dumb mistake on my part! Everything is working as expected. Nice work @sgrif

mcasper · 2015-12-21T01:32:54Z

diesel/src/migrations/migration.rs

+        let entry = t!(e);
+        let file_name = entry.file_name();
+        &file_name == "up.sql" || &file_name == "down.sql"
+    }) && t!(path.read_dir()).count() == 2


Should we potentially make this >= 2? In case some projects or companies want to be able to have other files in the dir, like example_output.txt?

I think this falls under the same umbrella as https://github.com/sgrif/diesel/pull/64/files#r48106745 -- I think it'll be fine to ignore files starting with ., but beyond that we want to make sure it contains exactly what's expected.

This is the first (small) piece of migration support, finding the directory containing migrations. Searches the current directory and its parents until it finds the directory containing Cargo.toml.

This still has a lot of rough edges, and isn't a fully finished feature, but it's pretty much the smallest step to something that I can test that passes. This adds the ability to run pending migrations (presumably from a build script). Migrations should be in the directory /migrations, where each migration is a timestamped sub-directory containing the files `up.sql` and `down.sql`. The migrations will be run, and an internal table will keep track of which migrations have or haven't been run. A lot of this code needs to be refactored.

Interestingly enough, the easiest way to accomplish this is to just use our own DSLs. We can't use codegen here, as that would create a circular dependency, but we just need to implement `Insertable` by hand. There were a few strange quirks that came as a result of this. I had to move `macros` up to the top so it could be used in `migrations`, which meant it could not do `use $crate::*;`. Additionally, we once again run into the bug where we can only use traits in public modules in our own crate.

We need to be sure that the migration running successfully and the insertion into `__diesel_schema_migrations` are an atomic operation. I wish I could write the generic `impl<T: From<Error>> From<TransactionError<T>>`...

This isn't a great place for it either, as we're universally printing to STDOUT, but it's marginally better than where it was before. I'm debating whether or not I should just inject a `Write` to the `Migration`.

We don't just want to have a bunch of directories with meaningless timestamps, as it becomes difficult to know what to edit/read. This also backfills some missing tests for pieces which can be tested in isolation.

There were many cases where we would try to run a migration from a directory that doesn't have the proper structure. This adds tests and changes the behavior to ensure everything is set up properly. What I really want to do is compare all the file names to `["up.sql", "down.sql"]`, but it seems to be more difficult to do that than what we have now.

This is already an implicit contract, so let's make it explicit through our types.

Concerns were raised about enforcing an integer here, and it seems like we'd prefer the flexibility of strings. While I was on the fence about this, I was recently told a story about someone trying to do millisecond precision on migrations in Rails that were generated by a script. This lead to `rake db:rollback` always rolling back the millisecond migration, since it won on integer ordering, even though it didn't win on string ordering. This sold me on just going with strings, even if it allows some strange behaviors.

Now that we've changed the version to be a string, rather than a number, I do want some sort of semantics about what is or isn't a migration, and what's valid in the same directory. With these rules, all files in the directory that aren't part of a migration must start with a `.`.

First pass at migrations

sgrif · 2015-12-25T19:24:03Z

PSA to those who tried this branch -- delete and recreate your database, there's some schema changes that won't work otherwise.

fables-tales reviewed Dec 20, 2015
View reviewed changes

mcasper reviewed Dec 21, 2015
View reviewed changes

sgrif force-pushed the sg-migrations branch 2 times, most recently from 53b389b to 98e28a9 Compare December 25, 2015 18:53

sgrif added 10 commits December 25, 2015 12:15

Add find_migration_directory

7593b42

This is the first (small) piece of migration support, finding the directory containing migrations. Searches the current directory and its parents until it finds the directory containing Cargo.toml.

Run migrations inside of a transaction

b395c02

We need to be sure that the migration running successfully and the insertion into `__diesel_schema_migrations` are an atomic operation. I wish I could write the generic `impl<T: From<Error>> From<TransactionError<T>>`...

Move reporting which migration is being run

412d7a3

This isn't a great place for it either, as we're universally printing to STDOUT, but it's marginally better than where it was before. I'm debating whether or not I should just inject a `Write` to the `Migration`.

Allow migrations to have more meaningful names

d01026f

We don't just want to have a bunch of directories with meaningless timestamps, as it becomes difficult to know what to edit/read. This also backfills some missing tests for pieces which can be tested in isolation.

Change migration version from string to bigint

d6cbf68

This is already an implicit contract, so let's make it explicit through our types.

sgrif force-pushed the sg-migrations branch from 98e28a9 to 7c83a4c Compare December 25, 2015 19:22

sgrif added a commit that referenced this pull request Dec 25, 2015

Merge pull request #64 from sgrif/sg-migrations

e705c79

First pass at migrations

sgrif merged commit e705c79 into master Dec 25, 2015

sgrif deleted the sg-migrations branch December 25, 2015 19:22

mfpiccolo mentioned this pull request Jan 26, 2016

diesel migration run panicked #140

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

First pass at migrations #64

First pass at migrations #64

sgrif commented Dec 20, 2015

sgrif commented Dec 20, 2015

fables-tales Dec 20, 2015

sgrif Dec 20, 2015

fables-tales Dec 20, 2015

mfpiccolo commented Dec 20, 2015

sgrif commented Dec 20, 2015

mfpiccolo commented Dec 20, 2015

mcasper Dec 21, 2015

sgrif Dec 21, 2015

sgrif commented Dec 25, 2015

First pass at migrations #64

First pass at migrations #64

Conversation

sgrif commented Dec 20, 2015

sgrif commented Dec 20, 2015

fables-tales Dec 20, 2015

Choose a reason for hiding this comment

sgrif Dec 20, 2015

Choose a reason for hiding this comment

fables-tales Dec 20, 2015

Choose a reason for hiding this comment

mfpiccolo commented Dec 20, 2015

sgrif commented Dec 20, 2015

mfpiccolo commented Dec 20, 2015

mcasper Dec 21, 2015

Choose a reason for hiding this comment

sgrif Dec 21, 2015

Choose a reason for hiding this comment

sgrif commented Dec 25, 2015