This repository has been archived by the owner. It is now read-only.

Accessing Mix tasks from release #67

Closed
jihaia opened this Issue Oct 11, 2014 · 52 comments

Comments

@jihaia

jihaia commented Oct 11, 2014

Sorry for the noob question - after I have packaged the release and deployed to new environment, I need to run some tasks in order to initialize the database etc. Are those MIX tasks accessible? is that the intent etc?

@bitwalker

This comment has been minimized.

Show comment
Hide comment
@bitwalker

bitwalker Oct 22, 2014

Owner

Since mix tasks aren't compiled into the project BEAMs, they won't be deployed with the release. In theory you could store them in priv, which is included in the release, and explicitly load them prior to execution. The main problem with that is that a lot of common functions in the Mix.* module namespace rely on knowing the current project directory, the build environment, etc. Not all of them of course, but you'd need to be very careful about how you write your tasks to ensure they work transparently between your dev environment and your release environment.

In general, mix tasks are intended for use during development, or more generally, when running your app from the project's source directory via mix or elixir. For code that you need to execute in production, perhaps conditionally, I would write compiled modules to perform those actions, and either call them during application start, or manually via remote shell.

If you have a specific example of something that you feel doesn't work well unless it's a mix task, let's take a look at it and see if there are some good alternatives, or if we can come up with a way to make your tasks work in a release.

Owner

bitwalker commented Oct 22, 2014

Since mix tasks aren't compiled into the project BEAMs, they won't be deployed with the release. In theory you could store them in priv, which is included in the release, and explicitly load them prior to execution. The main problem with that is that a lot of common functions in the Mix.* module namespace rely on knowing the current project directory, the build environment, etc. Not all of them of course, but you'd need to be very careful about how you write your tasks to ensure they work transparently between your dev environment and your release environment.

In general, mix tasks are intended for use during development, or more generally, when running your app from the project's source directory via mix or elixir. For code that you need to execute in production, perhaps conditionally, I would write compiled modules to perform those actions, and either call them during application start, or manually via remote shell.

If you have a specific example of something that you feel doesn't work well unless it's a mix task, let's take a look at it and see if there are some good alternatives, or if we can come up with a way to make your tasks work in a release.

@bitwalker bitwalker added the question label Oct 22, 2014

@bitwalker bitwalker self-assigned this Oct 22, 2014

@asaaki

This comment has been minimized.

Show comment
Hide comment
@asaaki

asaaki Oct 22, 2014

Contributor

👍 (GH is missing a Like button.)

Contributor

asaaki commented Oct 22, 2014

👍 (GH is missing a Like button.)

@bitwalker

This comment has been minimized.

Show comment
Hide comment
@bitwalker

bitwalker Nov 2, 2014

Owner

@jihaia Do you feel like that answers your question? Am I free to close this issue?

Owner

bitwalker commented Nov 2, 2014

@jihaia Do you feel like that answers your question? Am I free to close this issue?

@hahuang65

This comment has been minimized.

Show comment
Hide comment
@hahuang65

hahuang65 Mar 13, 2015

Stuff like mix ecto.migrate is pretty tough to do without access to mix tasks here. I've had to copy over my entire source folder to the server in order to run it.

Stuff like mix ecto.migrate is pretty tough to do without access to mix tasks here. I've had to copy over my entire source folder to the server in order to run it.

@shankardevy

This comment has been minimized.

Show comment
Hide comment
@shankardevy

shankardevy Mar 13, 2015

Contributor

+1 @hahuang65 . I am also in the same boat trying to execute ecto migrations and found no good solution than to dump my db locally and import it on the server. But may be, it's not what exrm is intended to do?

@bitwalker , is it ok to compare exrm to capistrano or ruby/rails community?

A capistrano task copies all the source files to server, runs the migration and starts the server (of course, through individual plugins for different tasks).

If exrm can be compared to capistrano, then it's expected that exrm gives a full solution (at least in future).

If exrm is not a deployment tool like capistrano, if it never was intended to solve this problem, then a new project needs to be created adding exrm as its dependency and continue the work where exrm left. Like, adding tasks to be run on server, running migrations, starting the server etc.

Contributor

shankardevy commented Mar 13, 2015

+1 @hahuang65 . I am also in the same boat trying to execute ecto migrations and found no good solution than to dump my db locally and import it on the server. But may be, it's not what exrm is intended to do?

@bitwalker , is it ok to compare exrm to capistrano or ruby/rails community?

A capistrano task copies all the source files to server, runs the migration and starts the server (of course, through individual plugins for different tasks).

If exrm can be compared to capistrano, then it's expected that exrm gives a full solution (at least in future).

If exrm is not a deployment tool like capistrano, if it never was intended to solve this problem, then a new project needs to be created adding exrm as its dependency and continue the work where exrm left. Like, adding tasks to be run on server, running migrations, starting the server etc.

@ericmj

This comment has been minimized.

Show comment
Hide comment
@ericmj

ericmj Mar 15, 2015

Contributor

Ecto should provide an easy to run migrations without mix tasks. It is possible today with Ecto.Migrator (please check the docs) but it should be as easy as running the mix task. Please open an issue on Ecto.

Contributor

ericmj commented Mar 15, 2015

Ecto should provide an easy to run migrations without mix tasks. It is possible today with Ecto.Migrator (please check the docs) but it should be as easy as running the mix task. Please open an issue on Ecto.

@josevalim

This comment has been minimized.

Show comment
Hide comment
@josevalim

josevalim Mar 16, 2015

@ericmj do you have any thoughts in mind? To me the plan is to exactly call Ecto.Migrator, it is public API and, if you are building releases, you can afford to call Ecto.Migrator with your options as part of your upgrade recipe.

@ericmj do you have any thoughts in mind? To me the plan is to exactly call Ecto.Migrator, it is public API and, if you are building releases, you can afford to call Ecto.Migrator with your options as part of your upgrade recipe.

@josevalim

This comment has been minimized.

Show comment
Hide comment
@josevalim

josevalim Mar 16, 2015

Maybe this is a matter of documentation but Ecto already does what needs to be done in order to support releases which is by providing a first-class Ecto.Migrator.

Maybe this is a matter of documentation but Ecto already does what needs to be done in order to support releases which is by providing a first-class Ecto.Migrator.

@bitwalker

This comment has been minimized.

Show comment
Hide comment
@bitwalker

bitwalker Mar 16, 2015

Owner

@josevalim Yeah I think this is one of the areas where custom appups are intended to be used, though that assumes you are using hot upgrades, rather than rolling restarts. Do you have any suggestions on a good general way of handling Ecto migrations?

@shankardevy exrm is definitely not trying to be the capistrano of Elixir, it's role is as a release packaging tool only - orchestrating deployment should be handled by tools appropriate to the user's given environment and needs, whether that's something like capistrano for Elixir, hand rolled scripts, or something else really varies case by case I think.

Owner

bitwalker commented Mar 16, 2015

@josevalim Yeah I think this is one of the areas where custom appups are intended to be used, though that assumes you are using hot upgrades, rather than rolling restarts. Do you have any suggestions on a good general way of handling Ecto migrations?

@shankardevy exrm is definitely not trying to be the capistrano of Elixir, it's role is as a release packaging tool only - orchestrating deployment should be handled by tools appropriate to the user's given environment and needs, whether that's something like capistrano for Elixir, hand rolled scripts, or something else really varies case by case I think.

@ericmj

This comment has been minimized.

Show comment
Hide comment
@ericmj

ericmj Mar 16, 2015

Contributor

@josevalim The Ecto.Migrator is already quite good actually, better than I thought. One possible improvement is to remove the directory option since that is configurable anyway.

Contributor

ericmj commented Mar 16, 2015

@josevalim The Ecto.Migrator is already quite good actually, better than I thought. One possible improvement is to remove the directory option since that is configurable anyway.

@HashNuke

This comment has been minimized.

Show comment
Hide comment
@HashNuke

HashNuke Jul 2, 2015

Contributor

Just realized that it's not possible to run migrations in a Phoenix app, when the app is running from a release.

My hackish way right now: pass an env var to turn on server: true in prod config when running mix release. This way other mix tasks can be used.

cc: @chrismccord

Contributor

HashNuke commented Jul 2, 2015

Just realized that it's not possible to run migrations in a Phoenix app, when the app is running from a release.

My hackish way right now: pass an env var to turn on server: true in prod config when running mix release. This way other mix tasks can be used.

cc: @chrismccord

@alexjp

This comment has been minimized.

Show comment
Hide comment
@alexjp

alexjp Jul 4, 2015

@HashNuke I am currently using exrm, where i can successfully and easily deploy a release with it on various machines. Only problem is migrations. Could you share your method ? As if I was a complete beginner with only a few months of elixir ? :)

alexjp commented Jul 4, 2015

@HashNuke I am currently using exrm, where i can successfully and easily deploy a release with it on various machines. Only problem is migrations. Could you share your method ? As if I was a complete beginner with only a few months of elixir ? :)

@bitwalker

This comment has been minimized.

Show comment
Hide comment
@bitwalker

bitwalker Jul 5, 2015

Owner

@HashNuke @alexjp If you have a solution that works, I'd like to document it and add it to exrm's docs :)

Owner

bitwalker commented Jul 5, 2015

@HashNuke @alexjp If you have a solution that works, I'd like to document it and add it to exrm's docs :)

@HashNuke

This comment has been minimized.

Show comment
Hide comment
@HashNuke

HashNuke Jul 5, 2015

Contributor

@bitwalker @alexjp Drafted a blog post yesterday about this and another thing (easier versioning). I'll link you to the blog post once I finish it.

The code I've been using to test is online. Here's how:

The SERVER=1 makes all the difference. The value doesn't have to be 1. Just set the SERVER env var to anything.

Contributor

HashNuke commented Jul 5, 2015

@bitwalker @alexjp Drafted a blog post yesterday about this and another thing (easier versioning). I'll link you to the blog post once I finish it.

The code I've been using to test is online. Here's how:

The SERVER=1 makes all the difference. The value doesn't have to be 1. Just set the SERVER env var to anything.

@josevalim

This comment has been minimized.

Show comment
Hide comment
@josevalim

josevalim Jul 5, 2015

If we are thinking about the problem generally, the biggest issue is that the scripting mode of mix does not really map to how we run apps in production with releases. For example, to run a task, we don't need to preload all modules as part of the release.

That's not to say there isn't a solution, quite the opposite, there are many ways to solve this:

  1. If you are building your release in the production machine, you can do things like: mix do release, ecto.migrate && run_release
  2. Allow options to be passed to the release executable, so we can do things like ./run_release --app phoenix:serve_endpoints true or pass extra config files ./run_release --config server and ./run_release --config migrator.
  3. Instead of running migrations in a separate node, you can just connect to a random node and execute the migrator thing.
  4. Ship with mix tasks inside the release and provide an easy way to run them via exrm. This is a bit cumbersome though because we likely want to use very few tasks and not all of them

I am not sure which one would be the best way to go because it depends on your workflow. If your plan is to build releases locally and just run them in production then we need to choose something like 2 or 3

Also, exrm as far as I know is not a replacement for capistrano, because it is about how to build your system, and not how to deploy it. Although bringing more deploy related features would likely streamline the process a lot (and help solve this problem). For example, if we pick 1 as our default workflow, it is easy to pass tasks to run before executing the release. @MSch may have some input on this.

If we are thinking about the problem generally, the biggest issue is that the scripting mode of mix does not really map to how we run apps in production with releases. For example, to run a task, we don't need to preload all modules as part of the release.

That's not to say there isn't a solution, quite the opposite, there are many ways to solve this:

  1. If you are building your release in the production machine, you can do things like: mix do release, ecto.migrate && run_release
  2. Allow options to be passed to the release executable, so we can do things like ./run_release --app phoenix:serve_endpoints true or pass extra config files ./run_release --config server and ./run_release --config migrator.
  3. Instead of running migrations in a separate node, you can just connect to a random node and execute the migrator thing.
  4. Ship with mix tasks inside the release and provide an easy way to run them via exrm. This is a bit cumbersome though because we likely want to use very few tasks and not all of them

I am not sure which one would be the best way to go because it depends on your workflow. If your plan is to build releases locally and just run them in production then we need to choose something like 2 or 3

Also, exrm as far as I know is not a replacement for capistrano, because it is about how to build your system, and not how to deploy it. Although bringing more deploy related features would likely streamline the process a lot (and help solve this problem). For example, if we pick 1 as our default workflow, it is easy to pass tasks to run before executing the release. @MSch may have some input on this.

@alexjp

This comment has been minimized.

Show comment
Hide comment
@alexjp

alexjp Jul 5, 2015

As i understand that this might be "bloating" exrm, solution 2 is exactly what i need.
It would be great to have migrations in the release, because from experience, when you separate db maintenance from code, sooner or later all hell breaks loose. (having db not in sync with what code expects it to be)

alexjp commented Jul 5, 2015

As i understand that this might be "bloating" exrm, solution 2 is exactly what i need.
It would be great to have migrations in the release, because from experience, when you separate db maintenance from code, sooner or later all hell breaks loose. (having db not in sync with what code expects it to be)

@HashNuke

This comment has been minimized.

Show comment
Hide comment
@HashNuke

HashNuke Jul 5, 2015

Contributor

@josevalim option-2 feels more friendlier. Being able to pass options to the release would be a huge goodie bag.

Contributor

HashNuke commented Jul 5, 2015

@josevalim option-2 feels more friendlier. Being able to pass options to the release would be a huge goodie bag.

@bitwalker

This comment has been minimized.

Show comment
Hide comment
@bitwalker

bitwalker Jul 6, 2015

Owner

@josevalim In Option 2, what is handling the options being sent to the executable? I can modify the boot script to pass options appended to -extra (currently --no-halt +iex is passed along with the console task, and foreground passes along any extra arguments to the release unaltered. In the case of the console task, IEx.CLI gets those options, but in the case of start or foreground, I'm not sure what happens to them, I assume they are accessible somehow, but I haven't poked around in that area at all.

Owner

bitwalker commented Jul 6, 2015

@josevalim In Option 2, what is handling the options being sent to the executable? I can modify the boot script to pass options appended to -extra (currently --no-halt +iex is passed along with the console task, and foreground passes along any extra arguments to the release unaltered. In the case of the console task, IEx.CLI gets those options, but in the case of start or foreground, I'm not sure what happens to them, I assume they are accessible somehow, but I haven't poked around in that area at all.

@josevalim

This comment has been minimized.

Show comment
Hide comment
@josevalim

josevalim Jul 6, 2015

@bitwalker it depends. If those configs are erlang configs, I assume you would also be able to pass them to the erl -config flag? It is worth mentioning that the options I mentioned above do not consider the exrm side (i.e. I am not sure how hard they are or if they are even possible to implement).

@bitwalker it depends. If those configs are erlang configs, I assume you would also be able to pass them to the erl -config flag? It is worth mentioning that the options I mentioned above do not consider the exrm side (i.e. I am not sure how hard they are or if they are even possible to implement).

@bitwalker bitwalker changed the title from QUESTION: Accessing Mix Tasks from Release to Accessing Mix tasks from release Aug 12, 2015

@olivermt

This comment has been minimized.

Show comment
Hide comment
@olivermt

olivermt Sep 7, 2015

I have solved this by just running mix ecto.migrate in my aws codedeploy scripts (since its indempotent anyways).

Is a writeup on that something that you guys would be interested in having as companion-documentation to this repository?

olivermt commented Sep 7, 2015

I have solved this by just running mix ecto.migrate in my aws codedeploy scripts (since its indempotent anyways).

Is a writeup on that something that you guys would be interested in having as companion-documentation to this repository?

@bitwalker

This comment has been minimized.

Show comment
Hide comment
@bitwalker

bitwalker Sep 8, 2015

Owner

@olivermt Definitely :)

Owner

bitwalker commented Sep 8, 2015

@olivermt Definitely :)

@olivermt

This comment has been minimized.

Show comment
Hide comment
@olivermt

olivermt Sep 8, 2015

Allright, will send pull request either next week or week after, a bit clogged up right now :)

olivermt commented Sep 8, 2015

Allright, will send pull request either next week or week after, a bit clogged up right now :)

@gdon

This comment has been minimized.

Show comment
Hide comment
@gdon

gdon Jan 2, 2016

Hi! Is there some plan to enable the Ecto migrate (and create) through run-app scritps in production?
As the sugestion number 2 from josevalim.
This would be very useful.
And thanks for the wonderful EXRM!

gdon commented Jan 2, 2016

Hi! Is there some plan to enable the Ecto migrate (and create) through run-app scritps in production?
As the sugestion number 2 from josevalim.
This would be very useful.
And thanks for the wonderful EXRM!

@bitwalker

This comment has been minimized.

Show comment
Hide comment
@bitwalker

bitwalker Jan 8, 2016

Owner

@gdon Currently there is no plans to support running Mix tasks with releases. My recommendation is to use the Ecto.Migrator API directly, and use configuration flags to determine whether or not to run the migrator. In my own apps, I run migrations every time the app boots, since they are idempotent.

Owner

bitwalker commented Jan 8, 2016

@gdon Currently there is no plans to support running Mix tasks with releases. My recommendation is to use the Ecto.Migrator API directly, and use configuration flags to determine whether or not to run the migrator. In my own apps, I run migrations every time the app boots, since they are idempotent.

@gdon

This comment has been minimized.

Show comment
Hide comment
@gdon

gdon Jan 8, 2016

Ok @bitwalker .
Thanks for adivise.

gdon commented Jan 8, 2016

Ok @bitwalker .
Thanks for adivise.

@iwarshak

This comment has been minimized.

Show comment
Hide comment
@iwarshak

iwarshak Jan 14, 2016

@bitwalker can you show how you are calling Ecto.Migrator?

@bitwalker can you show how you are calling Ecto.Migrator?

@zmoshansky

This comment has been minimized.

Show comment
Hide comment
@zmoshansky

zmoshansky Feb 3, 2016

@bitwalker can you show how you are calling Ecto.Migrator?
+1, I would also be very interested to see how Ecto migrations can be performed. Especially as part of hot upgrades.

@bitwalker can you show how you are calling Ecto.Migrator?
+1, I would also be very interested to see how Ecto migrations can be performed. Especially as part of hot upgrades.

@schurig

This comment has been minimized.

Show comment
Hide comment
@schurig

schurig Feb 11, 2016

@bitwalker can you show how you are calling Ecto.Migrator?

+1

schurig commented Feb 11, 2016

@bitwalker can you show how you are calling Ecto.Migrator?

+1

@schurig

This comment has been minimized.

Show comment
Hide comment
@schurig

schurig Feb 11, 2016

@zmoshansky @iwarshak @bitwalker

I found a way to run migrations when the server boots up:

# File: /lib/your_app.ex

Supervisor.start_link(children, opts)

becomes

res = Supervisor.start_link(children, opts)

path_to_migrations =
  case Mix.env do
    :dev  -> "priv/repo/migrations"
    _     -> "/home/apps/your-app/current/priv/repo/migrations"
  end
IO.puts "######### run migrations..."
Ecto.Migrator.run(YourApp.Repo, path_to_migrations, :up, all: true)

res

This doesn't feel right but it's the best approach I came up with so far - at least without having to include mix tasks in my release. Are there any better approaches?

schurig commented Feb 11, 2016

@zmoshansky @iwarshak @bitwalker

I found a way to run migrations when the server boots up:

# File: /lib/your_app.ex

Supervisor.start_link(children, opts)

becomes

res = Supervisor.start_link(children, opts)

path_to_migrations =
  case Mix.env do
    :dev  -> "priv/repo/migrations"
    _     -> "/home/apps/your-app/current/priv/repo/migrations"
  end
IO.puts "######### run migrations..."
Ecto.Migrator.run(YourApp.Repo, path_to_migrations, :up, all: true)

res

This doesn't feel right but it's the best approach I came up with so far - at least without having to include mix tasks in my release. Are there any better approaches?

@bitwalker

This comment has been minimized.

Show comment
Hide comment
@bitwalker

bitwalker Feb 11, 2016

Owner

@schurig That's exactly the right approach, but here's what I would change:

  • Reference priv/repo/migrations with Path.join(["#{:code.priv_dir(:yourapp)}", "repo", "migrations"]). This will work in all environments.
  • Running your migrations every time is fine, but you could also take the approach of passing a flag or setting an environment variable if you only want to run them conditionally. I prefer the approach of running them every time myself, but just so you are aware that there are some other options.
  • You should be running your migrations before starting your supervisor tree (if the migrations fail, your application shouldn't start).

/cc @zmoshansky @iwarshak

Owner

bitwalker commented Feb 11, 2016

@schurig That's exactly the right approach, but here's what I would change:

  • Reference priv/repo/migrations with Path.join(["#{:code.priv_dir(:yourapp)}", "repo", "migrations"]). This will work in all environments.
  • Running your migrations every time is fine, but you could also take the approach of passing a flag or setting an environment variable if you only want to run them conditionally. I prefer the approach of running them every time myself, but just so you are aware that there are some other options.
  • You should be running your migrations before starting your supervisor tree (if the migrations fail, your application shouldn't start).

/cc @zmoshansky @iwarshak

@schurig

This comment has been minimized.

Show comment
Hide comment
@schurig

schurig Feb 12, 2016

@bitwalker thank you!

I've tried to run my migrations before starting the supervisor tree but then it complains that the MyApp.Repo isn't started:

** (ArgumentError) repo MyApp.Repo is not started, please ensure it is part of your supervision tree

Current version:

res = Supervisor.start_link(children, opts)
IO.puts "######### run migrations..."
Ecto.Migrator.run(MyApp.Repo, Path.join(["#{:code.priv_dir(:my_app)}", "repo", "migrations"]), :up, all: true)
res

schurig commented Feb 12, 2016

@bitwalker thank you!

I've tried to run my migrations before starting the supervisor tree but then it complains that the MyApp.Repo isn't started:

** (ArgumentError) repo MyApp.Repo is not started, please ensure it is part of your supervision tree

Current version:

res = Supervisor.start_link(children, opts)
IO.puts "######### run migrations..."
Ecto.Migrator.run(MyApp.Repo, Path.join(["#{:code.priv_dir(:my_app)}", "repo", "migrations"]), :up, all: true)
res
@bitwalker

This comment has been minimized.

Show comment
Hide comment
@bitwalker

bitwalker Feb 12, 2016

Owner

@schurig Ah right, forgot about that. With that in mind, what you have is probably the best solution I can think of at the moment. If you end up finding a better approach, please let me know so I can document it!

Owner

bitwalker commented Feb 12, 2016

@schurig Ah right, forgot about that. With that in mind, what you have is probably the best solution I can think of at the moment. If you end up finding a better approach, please let me know so I can document it!

@MSch

This comment has been minimized.

Show comment
Hide comment
@MSch

MSch Feb 12, 2016

Contributor

Now that #286 is fixed I would recommend the following approach, which is how we're doing it:

Build an escript which invokes the tasks you want available in the release (using an exrm plugin: https://gist.github.com/MSch/9cfc185e24e2f3509650) and invoke it from the release:

rel/pssync/bin/pssync escript bin/release_tasks.escript migrate

This way we can control when and where to migrate, seed, etc.

There is some duplication/indirection since we want (most) of the tasks available both via mix in development and release_tasks.escript in production, but that is manageable. It also means we can migrate without fully starting our OTP application.

In an ideal future this would be upstreamed into exrm and more tightly integrated so that I can just do rel/pssync/bin/pssyc migrate but so far we've already had a great experience with our approach.

Contributor

MSch commented Feb 12, 2016

Now that #286 is fixed I would recommend the following approach, which is how we're doing it:

Build an escript which invokes the tasks you want available in the release (using an exrm plugin: https://gist.github.com/MSch/9cfc185e24e2f3509650) and invoke it from the release:

rel/pssync/bin/pssync escript bin/release_tasks.escript migrate

This way we can control when and where to migrate, seed, etc.

There is some duplication/indirection since we want (most) of the tasks available both via mix in development and release_tasks.escript in production, but that is manageable. It also means we can migrate without fully starting our OTP application.

In an ideal future this would be upstreamed into exrm and more tightly integrated so that I can just do rel/pssync/bin/pssyc migrate but so far we've already had a great experience with our approach.

@deybhayden

This comment has been minimized.

Show comment
Hide comment
@deybhayden

deybhayden Mar 15, 2016

@MSch Thanks for the comment and gist! Worked great for me - easy to expand for my purposes as well.

@MSch Thanks for the comment and gist! Worked great for me - easy to expand for my purposes as well.

@bitwalker

This comment has been minimized.

Show comment
Hide comment
@bitwalker

bitwalker Mar 17, 2016

Owner

I'll get this added to the documentation soon so it's easier to find.

Owner

bitwalker commented Mar 17, 2016

I'll get this added to the documentation soon so it's easier to find.

@killerswan

This comment has been minimized.

Show comment
Hide comment
@killerswan

killerswan Mar 29, 2016

Oooh, @MSch, thanks!

Oooh, @MSch, thanks!

@MSch

This comment has been minimized.

Show comment
Hide comment
@MSch

MSch Mar 30, 2016

Contributor

FYI @codermctwist @edgurgel @asaaki @killerswan We're now moving to bin/pssync command PS.ReleaseTasks migrate since #295 and #319 have been merged. You need exrm 1.0.3 for this.

Contributor

MSch commented Mar 30, 2016

FYI @codermctwist @edgurgel @asaaki @killerswan We're now moving to bin/pssync command PS.ReleaseTasks migrate since #295 and #319 have been merged. You need exrm 1.0.3 for this.

@olivermt

This comment has been minimized.

Show comment
Hide comment
@olivermt

olivermt Mar 30, 2016

@MSch could you post a code example of how you do migrations like that?

@MSch could you post a code example of how you do migrations like that?

@martin-langhoff

This comment has been minimized.

Show comment
Hide comment
@martin-langhoff

martin-langhoff Apr 5, 2016

Contributor

@MSch , @bitwalker so the recipe is to

  • pull out the logic out from the Mix Tasks to a module that does not depend on Mix
  • In dev, have Mix.Task.Foo call MyApp.Task.Foo
  • In prod, myapp.sh command MyApp.Task Foo

I like the approach, +1 to having it documented it somewhere :-)

Contributor

martin-langhoff commented Apr 5, 2016

@MSch , @bitwalker so the recipe is to

  • pull out the logic out from the Mix Tasks to a module that does not depend on Mix
  • In dev, have Mix.Task.Foo call MyApp.Task.Foo
  • In prod, myapp.sh command MyApp.Task Foo

I like the approach, +1 to having it documented it somewhere :-)

@martin-langhoff

This comment has been minimized.

Show comment
Hide comment
@martin-langhoff

martin-langhoff Apr 6, 2016

Contributor

It has been a dog to make this work, as any error results in uninformative failure in do_boot.

In the end, the right invocation ended up requiring an Elixir prefix, so on the command line I call:

rel/bin/myapp command Elixir.MyApp.DoSomething run
Contributor

martin-langhoff commented Apr 6, 2016

It has been a dog to make this work, as any error results in uninformative failure in do_boot.

In the end, the right invocation ended up requiring an Elixir prefix, so on the command line I call:

rel/bin/myapp command Elixir.MyApp.DoSomething run
@martin-langhoff

This comment has been minimized.

Show comment
Hide comment
@martin-langhoff

martin-langhoff Apr 6, 2016

Contributor

Hmm, any functions needing parameters fail. In this context, how do we get an args array built?

Contributor

martin-langhoff commented Apr 6, 2016

Hmm, any functions needing parameters fail. In this context, how do we get an args array built?

@jwarlander

This comment has been minimized.

Show comment
Hide comment
@jwarlander

jwarlander Apr 7, 2016

I couldn't figure out parameter passing either; I just arranged it so that I have a function for each purpose; eg:

bin/wolf command Elixir.Wolf.ReleaseTasks create
bin/wolf command Elixir.Wolf.ReleaseTasks migrate
bin/wolf command Elixir.Wolf.ReleaseTasks seed

My Wolf.ReleaseTasks module is directly modeled after the one @MSch posted about (https://gist.github.com/MSch/9cfc185e24e2f3509650), and can be found here:

https://gist.github.com/jwarlander/809deb2bb06c2c43abafd471591b2dea

The differences:

  • No main function, as I couldn't get parameters to work
  • Added create / drop to support initial deployment etc
  • Ended up having to call System.halt(0) at the end of each exposed function, or the command wouldn't finish

It feels like much of this could be made to be generic.

NOTE: No plugin needed w/ the command support that exists now.

I couldn't figure out parameter passing either; I just arranged it so that I have a function for each purpose; eg:

bin/wolf command Elixir.Wolf.ReleaseTasks create
bin/wolf command Elixir.Wolf.ReleaseTasks migrate
bin/wolf command Elixir.Wolf.ReleaseTasks seed

My Wolf.ReleaseTasks module is directly modeled after the one @MSch posted about (https://gist.github.com/MSch/9cfc185e24e2f3509650), and can be found here:

https://gist.github.com/jwarlander/809deb2bb06c2c43abafd471591b2dea

The differences:

  • No main function, as I couldn't get parameters to work
  • Added create / drop to support initial deployment etc
  • Ended up having to call System.halt(0) at the end of each exposed function, or the command wouldn't finish

It feels like much of this could be made to be generic.

NOTE: No plugin needed w/ the command support that exists now.

@martin-langhoff

This comment has been minimized.

Show comment
Hide comment
@martin-langhoff

martin-langhoff Apr 7, 2016

Contributor

My experience (and practice) matches that of @jwarlander ; however for some commands/actions I need to be able to pass parameters.

Contributor

martin-langhoff commented Apr 7, 2016

My experience (and practice) matches that of @jwarlander ; however for some commands/actions I need to be able to pass parameters.

@MSch

This comment has been minimized.

Show comment
Hide comment
@MSch

MSch Apr 7, 2016

Contributor

@martin-langhoff @jwarlander Look at the example code in the PR for how parameter passing works: #295

:init.get_plain_arguments() returns the arguments. I agree, it's rather annoying. also the results are Erlang charlists, so you have to do Enum.map(:init.get_plain_arguments(), &List.to_string/1)

One more thing, because bin/pssync command Elixir.PS.ReleaseTasks is really annoying we gave the module a non-Elixir namespaced name, that works this:

defmodule :sync_release do
end

Now we can do bin/pssync command sync_release migrate

Contributor

MSch commented Apr 7, 2016

@martin-langhoff @jwarlander Look at the example code in the PR for how parameter passing works: #295

:init.get_plain_arguments() returns the arguments. I agree, it's rather annoying. also the results are Erlang charlists, so you have to do Enum.map(:init.get_plain_arguments(), &List.to_string/1)

One more thing, because bin/pssync command Elixir.PS.ReleaseTasks is really annoying we gave the module a non-Elixir namespaced name, that works this:

defmodule :sync_release do
end

Now we can do bin/pssync command sync_release migrate

@MSch

This comment has been minimized.

Show comment
Hide comment
@MSch

MSch Apr 7, 2016

Contributor

It feels like much of this could be made to be generic.

I would love a way to register commands and then being able to do bin/<release_name> <command_name> <args> instead of having to do bin/<release_name> command <module> <function> <args>. But for us it's a low priority thing.

Ended up having to call System.halt(0) at the end of each exposed function, or the command wouldn't finish

Consider using :init.stop().

Contributor

MSch commented Apr 7, 2016

It feels like much of this could be made to be generic.

I would love a way to register commands and then being able to do bin/<release_name> <command_name> <args> instead of having to do bin/<release_name> command <module> <function> <args>. But for us it's a low priority thing.

Ended up having to call System.halt(0) at the end of each exposed function, or the command wouldn't finish

Consider using :init.stop().

@martin-langhoff

This comment has been minimized.

Show comment
Hide comment
@martin-langhoff

martin-langhoff May 3, 2016

Contributor

For the record, I have used @MSch 's advice and it's worked beautifully.

there are some issues still on v1.0.3 around boot_clean vs boot that I cannot quite figure out; more specifically #340 .

Contributor

martin-langhoff commented May 3, 2016

For the record, I have used @MSch 's advice and it's worked beautifully.

there are some issues still on v1.0.3 around boot_clean vs boot that I cannot quite figure out; more specifically #340 .

@bitwalker

This comment has been minimized.

Show comment
Hide comment
@bitwalker

bitwalker May 25, 2016

Owner

Just to follow up regarding #340, I pushed a fix in 1.0.5 for the console_clean command.

Owner

bitwalker commented May 25, 2016

Just to follow up regarding #340, I pushed a fix in 1.0.5 for the console_clean command.

@martin-langhoff

This comment has been minimized.

Show comment
Hide comment
@martin-langhoff

martin-langhoff May 25, 2016

Contributor

Yes I can confirm that on 1.0.5 this works great. Thank you!

One thing I have ended up needing for mnesia DB prep and seeding is a function that mimics what Application.ensure_all_started() does, except that it only starts the deps, not the app itself.

Contributor

martin-langhoff commented May 25, 2016

Yes I can confirm that on 1.0.5 this works great. Thank you!

One thing I have ended up needing for mnesia DB prep and seeding is a function that mimics what Application.ensure_all_started() does, except that it only starts the deps, not the app itself.

@martin-langhoff

This comment has been minimized.

Show comment
Hide comment
@martin-langhoff

martin-langhoff May 25, 2016

Contributor

So I declare

  @shortdoc "Start all deps without starting the app itself"
  def ensure_all_deps_started() do
    Application.load(:foo)
    Enum.map(Application.spec(:foo, :applications), &Application.load(&1))
    Enum.map(Application.spec(:foo, :applications), &Application.ensure_all_started(&1))
  end

So then in the call I expose for command, I do something like:

def run_and_exit() do
   foo.ensure_all_deps_started
   foo.prepare_or_upgrade_stuff
   :init.stop
end
Contributor

martin-langhoff commented May 25, 2016

So I declare

  @shortdoc "Start all deps without starting the app itself"
  def ensure_all_deps_started() do
    Application.load(:foo)
    Enum.map(Application.spec(:foo, :applications), &Application.load(&1))
    Enum.map(Application.spec(:foo, :applications), &Application.ensure_all_started(&1))
  end

So then in the call I expose for command, I do something like:

def run_and_exit() do
   foo.ensure_all_deps_started
   foo.prepare_or_upgrade_stuff
   :init.stop
end
@martin-langhoff

This comment has been minimized.

Show comment
Hide comment
@martin-langhoff

martin-langhoff May 25, 2016

Contributor

Formatting of complex command args is still terrible. Might have a patch to propose soon.

Contributor

martin-langhoff commented May 25, 2016

Formatting of complex command args is still terrible. Might have a patch to propose soon.

@jbsmith

This comment has been minimized.

Show comment
Hide comment
@jbsmith

jbsmith May 26, 2016

one additional item to note:

To get escripts to run I had to set the ENV var for ERL_LIBS
e.g.
ERL_LIBS = /opt/app/rel/MYAPP/lib /opt/app/rel/MYAPP/bin/MYAPP escript bin/my_release_tasks.escript args

without that, lots of issues with do_boot or not able to find modules etc.

with the foreground option, that variable is normally set, but for escript and command it is not.

jbsmith commented May 26, 2016

one additional item to note:

To get escripts to run I had to set the ENV var for ERL_LIBS
e.g.
ERL_LIBS = /opt/app/rel/MYAPP/lib /opt/app/rel/MYAPP/bin/MYAPP escript bin/my_release_tasks.escript args

without that, lots of issues with do_boot or not able to find modules etc.

with the foreground option, that variable is normally set, but for escript and command it is not.

@martin-langhoff

This comment has been minimized.

Show comment
Hide comment
@martin-langhoff

martin-langhoff May 27, 2016

Contributor

Related #354 #359 for rpc args parsing.

I am of the opinion that rpc and command should auto-format multiple args. For example

rpc module function arg1 arg2 arg3`

... should be transformed into

rpc module function '[<<"arg1">>,<<"arg2">>,<<"arg3">>].'

so that functions that take in parameters work similarly under Mix and exrm command invocations. To avoid mangling already-formatted inputs, it would only kick in if there's more than one parameter and the group does not end in '.'.

Admittedly, it is a bit of DWIM.

@bitwalker - thoughts?

Contributor

martin-langhoff commented May 27, 2016

Related #354 #359 for rpc args parsing.

I am of the opinion that rpc and command should auto-format multiple args. For example

rpc module function arg1 arg2 arg3`

... should be transformed into

rpc module function '[<<"arg1">>,<<"arg2">>,<<"arg3">>].'

so that functions that take in parameters work similarly under Mix and exrm command invocations. To avoid mangling already-formatted inputs, it would only kick in if there's more than one parameter and the group does not end in '.'.

Admittedly, it is a bit of DWIM.

@bitwalker - thoughts?

@noma4i noma4i referenced this issue Feb 22, 2017

Closed

Stop using Mix.env #13

2 of 2 tasks complete

cjbottaro referenced this issue in cjbottaro/faktory_worker_ex Dec 28, 2017

vanderhoop added a commit to stride-nyc/remote_retro that referenced this issue Mar 31, 2018

Run migrations on application startup
  - Gigalixir leverages distillery, and we don't have access to mix tasks in distillery releases
  - solution found at:
    - bitwalker/exrm#67 (comment)
      and
    - bitwalker/exrm#67 (comment)

@bitwalker bitwalker closed this Apr 6, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.