Move migration API from the diesel-cli crate into another diesel crate

[vityafx]

I am packaging my app into a docker image. So, it is packaged but I must perform migrations for making it working correctly. I have to use the diesel-cli binary crate but the docker image with it is 1.5GB in size. It is really wasteful because I need 1.5GB only for performing one single command (of diesel-cli). Everyone would prefer a single diesel-cli docker image but it is impossible to create it at this moment (it can't be compiled with musl yet). The other option and actually the one which is better - move out the migration-related code from the diesel-cli into separate crate, so the rust application can depend on it and call the function right in the beginning of its main function. After that the diesel-cli crate could depend on that new diesel-XXX crate and simply call its functions.

[vityafx]

Probably this is it: https://docs.rs/diesel_migrations/1.1.0/diesel_migrations/macro.embed_migrations.html

[brandur]

For what it's worth, I ran into the same problem while trying to build an app, and embedded migrations turned out to be an adequate solution for me.

Here's my code in case it's helpful:

// Migrations get pulled into the final binary. This makes it quite a bit
// easier to run them on remote clusters without trouble.
embed_migrations!("./migrations");

...

fn subcommand_migrate(log: &Logger, matches: &ArgMatches, options: &GlobalOptions) -> Result<()> {
    let _matches = matches.subcommand_matches("migration").unwrap();
    let conn = connection(log)?;

    info!(log, "Running migrations");

    if options.quiet {
        embedded_migrations::run(&*conn)
    } else {
        embedded_migrations::run_with_output(&*conn, &mut std::io::stdout())
    }.chain_err(|| "Error running migrations")?;

    info!(log, "Finished migrations");
    Ok(())
}

It has an advantage compared to containerizing diesel-cli in that because I just run them as a subcommand of my main program, I don't need to maintain a separate Docker image, and can run my migrations by attaching to any existing container.

The downside is that down migrations, or any kind of finer migration operations (rolling back/forward by steps or anything else) aren't supported. This is usually fine given that my migrations are well-vetted before going to prod and in case of an emergency I still have a psql console, but having the option would probably be better.

[vityafx]

@brandur Yes, I am already trying this right now. But I am gonna perform migrations silently, without needing for explicit command call, so you just run the app in the docker image and the database is set up. Thanks for your response, now I am sure this is it.

This issue would not be created if the diesel had all of this great stuff documented. The project is so big and is so complex. A lot of time I just have to go to this repository and try searching in the code.

[sgrif]

This issue would not be created if the diesel had all of this great stuff documented.

You did literally link to the documentation for it one comment ago. ;)

[vityafx]

@brandur How do you compile it in the rust-musl-builder?

   Compiling diesel_migrations v1.1.0
error: /home/rust/src/target/release/deps/libmigrations_macros-8295541ac8c86046.so: undefined symbol: SSL_set_ex_data
  --> /home/rust/.cargo/registry/src/github.com-1ecc6299db9ec823/diesel_migrations-1.1.0/src/lib.rs:77:1
   |
77 | extern crate migrations_macros;
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

error: Could not compile `diesel_migrations`.
warning: build failed, waiting for other jobs to finish...
error: build failed
The command '/bin/sh -c cargo build --release' returned a non-zero code: 101
time="2018-03-09T21:21:28Z" level=fatal msg="exit status 101"

Or, hm, you dont..

Interesting thing:

# nm /usr/lib/x86_64-linux-gnu/libssl.a | grep SSL_set_ex
00000000000048f0 T SSL_set_ex_data

@sgrif Yes, but before that I had to read a part of source code. This crate was not mentioned on the https://diesel.rs or diesel docs.

[brandur]

@vityafx Mine seems to work well. Maybe make sure that you follow all the caveats in the project's README. For example, it's suggested that this be added to Cargo.toml to make Diesel work:

[dependencies]
# This is needed to make sure that Cargo statically links against
# `libssl`. This should happen automatically, but it doesn't.
openssl-sys = "0.9"

[patch.crates-io]
# This is needed to handle cross-compilation of libpq.
pq-sys = { git = 'https://github.com/golddranks/pq-sys' }

If it helps, here's my completed Dockerfile, but as you'll see, there's nothing special going on in there.

You did literally link to the documentation for it one comment ago. ;)

@sgrif I'm going to second that although it's really nice that some of this documentation does turn out to exist, it's not very discoverable. I almost always find things by ag'ing the source code and then using that path to go back to docs.rs to look at rendered docs.

[vityafx]

@brandur I have done that already without success.

[brandur]

@vityafx Based on your compilation error, it seems that somehow OpenSSL might be absent from the image or something. Unfortunately I'm cargo culting to a large degree myself and not really an expert, but if you're really stuck, I'd suggest going back to the beginning and trying to get it configured again from scratch. If you'd like, you can use by Dockerfile and .dockerignore as a template.

[vityafx]

@brandur I have doubly checked that the library is there and it contains the needed symbol. I am stuck because everything seems to be okay but it is not. Thanks for the help anyway.