escript-related mix tasks #2974

alco · 2014-12-30T20:24:12Z

This PR implements a few additional escript-related mix tasks, outlined in this proposal.

Technical details

mix escript.install has been added that works similarly to mix archive.install: it takes a path or a URL to an arbitrary file and puts it under ~/.mix/escripts.
mix escript lists all executable files under ~/.mix/escripts.
mix escript.uninstall removes the file with the given name from ~/.mix/escripts.

Additional concerns

the ability to escript.install a prebuilt escript has been added because archive.install could already do that for .ez files, and it makes even more sense to have this work with escripts as they are not dependent on the installed Elixir version.
there is no environment variable MIX_ESCRIPTS because, as I mentioned, escript are generally independent of the installed Elixir version.

TODO

tests
to keep archive.install and escript.install symmetric, both should be able to fetch repositories and hex packages. The package will be fetched into a temporary directory, in the case of an escript it will also fetch dependencies. Then everything is compiled and subsequently archive.install or escript.install is called with the resulting archive or escript file. This makes it possible to host mix tasks on hex.pm too.
the command-line invocation syntax is just an idea. I have considered lots of alternatives, but would like to hear what the team thinks about it first. We can't really say that it is similar to the dep syntax in mix.exs because that would mean having something like this:
```
mix escript.install phoenix
mix escript.install phoenix 1.0.2
mix escript.install phoenix github phoenixframework/phoenix
```
I would much rather have the Mix task infer the source type from the URL given to it. The biggest problem is then to distinguish a local path from the package name.
if we allow both archives and escripts to be fetched from hex.pm, it may be confusing in which way a given package is supposed to be used: as a dependency, as an archive, or as an escript. Perhaps, hex.pm could indicate that visually in some way, or via the project's name.

josevalim · 2014-12-31T09:14:13Z

@alco Great work! I would like to suggest though to add the hex functionality in another pull request because I am considering backporting escript install to 1.0.x. I will add some specific comment to the PR overall soon.

josevalim · 2014-12-31T09:18:33Z

the ability to escript.install a prebuilt escript has been added because archive.install could already do that for .ez files, and it makes even more sense to have this work with escripts as they are not dependent on the installed Elixir version.

Perfect.

there is no environment variable MIX_ESCRIPTS because, as I mentioned, escript are generally independent of the installed Elixir version.

Makes sense! And MIX_HOME would already change it.

if we allow both archives and escripts to be fetched from hex.pm, it may be confusing in which way a given package is supposed to be used: as a dependency, as an archive, or as an escript. Perhaps, hex.pm could indicate that visually in some way, or via the project's name.

Honestly, it is very likely you don't want to install something as an archive. It will break with Elixir version, it is global and therefore has a chance of conflicting with your code, etc. Maybe we should simply not add this functionality to archives? I know they won't be symmetric then... but it will be for good reasons?

One final consideration: should we check if the ~/.mix/escripts is in your path after you install an escript and emit a warning if it is not? We should also update the windows installer to automatically add it to your $PATH and add a note to the homebrew one.

josevalim · 2014-12-31T09:24:07Z

lib/mix/lib/mix/tasks/escript.build.ex

@@ -108,7 +108,7 @@ defmodule Mix.Tasks.Escript.Build do
  defp escriptize(project, language, force, should_consolidate) do
    escript_opts = project[:escript] || []

-    script_name  = to_string(escript_opts[:name] || project[:app])
+    script_name  = Mix.Escript.escript_name(project)


Should we allow the -o option as in archive.build?

I don't think it's useful for escripts. archive.build works with non-mix projects according to its docs. Escripts require mix.exs, so it would be simpler to have only one place where the name can be changed – in mix.exs.

Good point, thanks!

alco · 2014-12-31T18:40:49Z

Honestly, it is very likely you don't want to install something as an archive. It will break with Elixir version, it is global and therefore has a chance of conflicting with your code, etc. Maybe we should simply not add this functionality to archives? I know they won't be symmetric then... but it will be for good reasons?

Currently there are a few inconveniences related to how archives can be installed. First, the author has to build an archive and put it somewhere online for it to be installable with mix archive.install <url>. Second, if there is no .ez file hosted online, the user will need to clone the repository, build the archive (possibly changing the tag to match their Elixir version) and then run mix archive.install.

If mix archive.install supported fetching from hex.pm, then the author would only need to push to hex.pm, and the user would only need to mix archive.install <package name>. And if hex.pm kept track of Elixir version requirement for each package version, it could print warnings or automatically resolve to the supported package version.

In that sense, I see enabling support for packages in mix archive.install to have only benefits. Of course, archives remain brittle in the face of Elixir version updates. But if we allow to install them at all, installing from a hex package makes more sense than what we have now.

I like to have things like mix docs and mix dialyze installed system-wide so that they can be used with any project without prior configuration. They could be replaced with escripts, but since Mix tasks need to interact with the source code, they are already tied to a particular Elixir version. I know that your preference is to add Mix tasks as dependencies. We could go further and remove mix install.archive altogether. But if it remains, it should be able to fetch hex packages and, maybe, fetch repositories automatically.

josevalim · 2014-12-31T20:25:17Z

I like to have things like mix docs and mix dialyze installed system-wide so that they can be used with any project without prior configuration.

Right, that's the only reasonable use case. Even though, it can cause issues: what if we have ex_doc installed as an archive and a project list ex_doc as a dependency? Are we currently guaranteeing the package one has higher preference? Furthermore, depending on the order things run, I could end up using both versions like this:

We run mix docs
Since mix docs is in an archive, we start running it
Because mix docs need to compile your code, it will fetch the dependencies and load the ex_doc version in the project
mix docs fail because of different module versions :(

So there are a bunch of issues we should fix before really promoting archives. :(

josevalim · 2014-12-31T20:25:41Z

Also, thanks for the discussion: ❤️

ericmj · 2015-01-01T15:44:52Z

lib/mix/lib/mix/tasks/escript.install.ex

+      mix escript.install github.com/example/foo             # fetches git://github.com/example/example.git
+      mix escript.install https://github.com/example/foo     # fetches https://github.com/example/example.git
+      mix escript.install --scm=git example.com/foo          # fetches git://example.com/example
+      mix escript.install --scm=git https://example.com/foo  # fetches https://example.com/example


We should not infer git from a http url. It limits http urls to the git SCM but a lot of SCMs allow using http as transport.

I still vote for the syntax proposed in the mailing list.

In the case of https://example.com/foo.git above, Git would be inferred from the trailing .git part.

After some consideration I see the appeal of sticking with your proposal because of its similarity with dep syntax. It doesn't define a few things though. What is your take on these cases?

escript.install ./local_file # ./ is used to distinguish from a hex package escript.install path/to/local_file # cannot be a hex package, so it can imply a local path escript.install path path/to/local_repo # ugly # What about this? Should we forbid fetching remote scripts? escript.install ??? https://example.com/escript_file

I realize that in the common case hex packages should be preferred, but we still have to define what happens with the other cases.

The ambiguity of supporting a prebuilt escript could be solved with a dedicated task:

escript.install-file local_file escript.install-file path/to/local_file escript.install-file http://example.com/escript # or a flag escript.install --file local_file # etc...

It may not be all too important to be able to install prebuilt escripts, but leaving it out makes it feel incomplete.

What do you think, @josevalim, @ericmj ?

If we are going to support prebuilt escripts I prefer these of your proposals:

escript.install ./local_file escript.install path/to/local_file escript.install https://example.com/escript_file

The following would build and install a local project (i.e. not a prebuilt file):

escript.install path path/to/local_repo

I like @ericmj's proposal here. With no arguments it means to install prebuilt scripts, if you pass, hex|path|git, it means installing from a project (which therefore requires building and mix).

I definitely think though we should move this into another discussion, as we can ship prebuilt installation earlier. :)

I like @ericmj's proposal here. With no arguments it means to install prebuilt scripts, if you pass, hex|path|git, it means installing from a project (which therefore requires building and mix).

So have we settled on the following syntax?

# this PR escript.install X # X is a path or a URL to a prebuilt escript file # future PR escript.install X Y [Z] # X is the package type (hex|path|git), # Y is the package name or URL # Z would be a version for a Hex package, or a ref for a Git repo

In that case, we can drop the ./ prefix when installing a file from the local directory. It was used to disambiguate from the mix escript.install <hex-package> syntax.

I definitely like it. @alco, it is not clear if you like it though. Do you?

I like it too. Will push an update tonight.

alco · 2015-01-01T21:36:48Z

So there are a bunch of issues we should fix before really promoting archives. :(

@josevalim right, archives feel like they haven't been thought out very well. I'm not going to add any changes to them in this PR, but I will probably open another issue for them.

alco · 2015-01-20T23:02:16Z

Pushed an update that only implements installation of prebuilt escripts. Did some refactoring, added tests.

josevalim · 2015-09-13T11:47:05Z

@alco can you please rebase? Let's start work on this front. I have reviewed your changes and they look great, although I would try to unity the installer for both. It seems the only different is the path and that escripts need to call chmod? Thank you!

alco · 2015-09-14T09:30:33Z

I'll rebase later today, there are some conflicts I need to have a closer look at.

The differences in installing escripts and archives go beyond just having to call chmod. For archives we also check the Elixir version and forcibly remove previous ones. If you want to have a single main installation routine, it will need to have the ability to customize what to run before and after it.

josevalim · 2015-09-14T09:58:05Z

I was thinking this routing is the same https://github.com/elixir-lang/elixir/pull/2974/files#diff-2c79c79704bca75ad6388315cad90838R37 up to the should_install?. From that moment on, we can likely pass an anonymous function that would finish the job.

alco · 2015-09-15T07:39:32Z

Rebased. The biggest change I had to make during the rebase process was replacing Mix.Utils.copy_path! with Mix.Utils.read_path which I copied almost verbatim from archive.install. You can see it in the last commit. Feel free to make any further changes.

josevalim · 2015-09-15T07:56:58Z

@alco awesome! Can you please give a try at unifying both installation approaches? It will be specially important once we start building archives and escripts from git, so the sooner we unify, the best.

Also, what if we rename Mix.Local.Utils to Mix.Local.Installer?

lexmag · 2015-09-15T07:59:57Z

lib/mix/lib/mix/local_utils.ex

@@ -0,0 +1,70 @@
+defmodule Mix.Local.Utils do


Shouldn't be mix/local/utils.ex used as a name for a file?

It is still a valid question.

alco · 2015-09-15T08:37:52Z

It's unlikely that I'll have time to work on this before the weekend, sorry. Feel free to take it or I'll get back to you next week.

josevalim · 2015-09-15T08:46:42Z

@alco oh, no worries at all. I forgot to say but this is not for 1.1, it will be for 1.2 or 1.3, which means we have at least 2 months ahead of us. :)

alco · 2015-09-15T08:52:56Z

OK. Are you planning to include install-from-git and/or install-from-hex in the same version?

josevalim · 2015-09-15T08:59:20Z

@alco it is your call. but given that we have some time, it would be nice to finally make progress on this (I am aware it stalled mostly due to me being unavailable though, sorry).

alco · 2015-12-30T15:15:44Z

Oh wow, this PR is more than a year old. Time flies X_X

Looking into it now.

alco · 2015-12-30T15:41:59Z

The latest Travis build has two issues that look unrelated to the changes in this PR:

15:17:51.138 [error] GenEvent handler GenEventTest.ReplyHandler installed in #PID<0.7394.0> terminating
** (RuntimeError) oops
    test/elixir/gen_event_test.exs:44: GenEventTest.ReplyHandler.handle_event/2
    (elixir) lib/gen_event.ex:1101: GenEvent.do_handler/3
    (elixir) lib/gen_event.ex:942: GenEvent.server_update/5
    (elixir) lib/gen_event.ex:927: GenEvent.server_notify/7
    (elixir) lib/gen_event.ex:900: GenEvent.server_event/4
    (elixir) lib/gen_event.ex:696: GenEvent.handle_msg/5
    (stdlib) proc_lib.erl:239: :proc_lib.init_p_do_apply/3
Last message: :raise
State: #PID<0.7393.0>

and

  1) test get and compile dependencies for rebar3 (Mix.RebarTest)
     test/mix/rebar_test.exs:179
     ** (RuntimeError) no shell process input given for yes?/1
     stacktrace:
       (mix) lib/mix/shell/process.ex:155: Mix.Shell.Process.yes?/1
       (mix) lib/mix/tasks/deps.compile.ex:146: Mix.Tasks.Deps.Compile.handle_rebar_not_found/1
       (mix) lib/mix/tasks/deps.compile.ex:130: Mix.Tasks.Deps.Compile.do_rebar3/2
       (mix) lib/mix/tasks/deps.compile.ex:66: anonymous fn/3 in Mix.Tasks.Deps.Compile.compile/1
       (elixir) lib/enum.ex:1088: Enum."-map/2-lists^map/1-0-"/2
       (mix) lib/mix/tasks/deps.compile.ex:53: Mix.Tasks.Deps.Compile.compile/1
       test/mix/rebar_test.exs:186: anonymous fn/0 in Mix.RebarTest.test get and compile dependencies for rebar3/1
       (elixir) lib/file.ex:1138: File.cd!/2
       test/mix/rebar_test.exs:182

josevalim · 2015-12-30T15:43:23Z

Master is passing. Is it a race condition or something wrong on rebase?

alco · 2015-12-30T15:49:26Z

The rebase was clean, without conflicts. I don't see those errors locally.

alco · 2015-12-30T21:49:44Z

@josevalim I did some unification, still WIP. Will continue tomorrow.

alco · 2015-12-31T12:39:11Z

It's green now 🤷. I wasn't able to reproduce the failure ** (RuntimeError) no shell process input given for yes?/1 mentioned above.

alco · 2015-12-31T12:42:18Z

@josevalim This is ready for review.

josevalim · 2016-02-21T12:28:43Z

lib/mix/lib/mix/archive.ex

@@ -9,21 +9,6 @@ defmodule Mix.Archive do
  """

  @doc """
-  Returns the archive name based on `app` and `version`.


This is backwards incompatible. We need to keep it here.

josevalim · 2016-02-21T12:32:46Z

@alco thank you! I have added just one tiny comment, everything else is perfect (and we need to tidy up the FIXMEs in the code).

Also, we need to make sure this works on Windows because on Windows you need to use escript path/to/escript to run them: On Windows, we will probably need to use .bat files. See here: http://stackoverflow.com/questions/26569257/how-to-run-an-elixir-escript-on-windows-8-1

So I think we should likely generate .bat files alongside the escript files and just invoke them. Can you check how this would work on Windows? If for some reason you don't have a Windows machine around, let me know and we can merge this and I can tidy it up later on.

…hive.install'

The message isn't handled by install() because it raises an exception prior to asking for confirmation.

alco · 2016-02-24T13:51:43Z

@josevalim I have restored Mix.Archive.name/2 and removed that FIXMEs. Also rebased on master.

I don't have access to a Windows machine at the moment.

josevalim · 2016-02-25T16:12:09Z

lib/mix/lib/mix/local.ex

+  end
+
+  def path_for(:escript) do
+    Path.join(Mix.Utils.mix_home, "escripts")


Should we provide MIX_ESCRIPTS?

Considering we have MIX_ARCHIVES 👍.

This has already been asked and answered here (see the second quote).

escript-related mix tasks

josevalim · 2016-02-25T16:36:00Z

lib/mix/lib/mix/tasks/escript.install.ex

+
+  def after_install(dst, _previous) do
+    File.chmod!(dst, @escript_file_mode)
+    check_discoverability(dst)


I would like to check this is actually a escript. Otherwise this command can be used to install any executable and it may be a bad idea.

josevalim reviewed Dec 31, 2014
View reviewed changes

ericmj reviewed Jan 1, 2015
View reviewed changes

alco force-pushed the alco/escript.install branch from a8ce248 to 285fb81 Compare January 20, 2015 23:01

alco changed the title ~~[WIP] escript-related mix tasks~~ escript-related mix tasks Jan 24, 2015

5thWall mentioned this pull request Aug 29, 2015

Create an easily installable binary to go in $PATH lpil/dogma#18

Closed

alco force-pushed the alco/escript.install branch from 285fb81 to 842a9b7 Compare September 15, 2015 07:37

lexmag reviewed Sep 15, 2015
View reviewed changes

alco force-pushed the alco/escript.install branch from 842a9b7 to e6142e7 Compare December 30, 2015 15:15

alco force-pushed the alco/escript.install branch from 7075916 to a662122 Compare December 31, 2015 12:34

josevalim added this to the v1.3.0 milestone Dec 31, 2015

josevalim reviewed Feb 21, 2016
View reviewed changes

alco added 15 commits February 24, 2016 15:43

Implement 'mix escript' task

75ee876

Implement 'mix escript.uninstall' task

99d1d84

Implement 'mix escript.install' task that works similarly to 'mix arc…

54c4f7a

…hive.install'

Fix an issue with removing a non-existing escript

98ee543

Check whether the installed escript is in the PATH after installation

7e12b20

Check the file type when building the list of installed escripts

4dd23d6

Extract common code for archive- and escript-related tasks

4a494b9

Add tests for the new escript tasks

504306b

Rewrite escript.install using Mix.Utils.read_path instead of copy_path!

594aca1

Fix a typo in the name of MIX_ARCHIVES in the docs.

eee5631

Rename Mix.Local.Utils to Mix.Local.Installer

f07ca71

Fix some style issues

91e4f41

Unify the common code involved in installing archives and escripts

f5410d4

Remove a redundant message send in a mix archive test

819d1c5

The message isn't handled by install() because it raises an exception prior to asking for confirmation.

Remove an outdated FIXME commit

2b417ae

alco force-pushed the alco/escript.install branch from a662122 to 2b417ae Compare February 24, 2016 13:50

alco changed the title ~~[WIP] escript-related mix tasks~~ escript-related mix tasks Feb 24, 2016

josevalim reviewed Feb 25, 2016
View reviewed changes

josevalim added a commit that referenced this pull request Feb 25, 2016

Merge pull request #2974 from alco/alco/escript.install

25ca9c0

escript-related mix tasks

josevalim merged commit 25ca9c0 into elixir-lang:master Feb 25, 2016

josevalim reviewed Feb 25, 2016
View reviewed changes

alco deleted the alco/escript.install branch February 26, 2016 13:39

escript-related mix tasks #2974

escript-related mix tasks #2974

Uh oh!

Conversation

alco commented Dec 30, 2014

Technical details

Additional concerns

TODO

Uh oh!

josevalim commented Dec 31, 2014

Uh oh!

josevalim commented Dec 31, 2014

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alco commented Dec 31, 2014

Uh oh!

josevalim commented Dec 31, 2014

Uh oh!

josevalim commented Dec 31, 2014

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alco commented Jan 1, 2015

Uh oh!

alco commented Jan 20, 2015

Uh oh!

josevalim commented Sep 13, 2015

Uh oh!

alco commented Sep 14, 2015

Uh oh!

josevalim commented Sep 14, 2015

Uh oh!

alco commented Sep 15, 2015

Uh oh!

josevalim commented Sep 15, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alco commented Sep 15, 2015

Uh oh!

josevalim commented Sep 15, 2015

Uh oh!

alco commented Sep 15, 2015

Uh oh!

josevalim commented Sep 15, 2015

Uh oh!

alco commented Dec 30, 2015

Uh oh!

alco commented Dec 30, 2015

Uh oh!

josevalim commented Dec 30, 2015

Uh oh!

alco commented Dec 30, 2015

Uh oh!

alco commented Dec 30, 2015

Uh oh!

alco commented Dec 31, 2015