mix archive.install fails in fetching optional dependencies

[Eiji7]

Environment

$ asdf current
elixir         1.6.4-otp-20 (set by …) # checked also ref:master
erlang         20.3.2  (set by …)

Funtoo Linux current x86-64bit on intel64-ivybridge

Current behavior

Installing archives from hex or git/github source fails in fetching optional dependencies.

For example inch_ex does not fails, but ecto or phoenix fails.

Expected behavior

All archives should be installed.

Example error

mix archive.install hex phoenix
Resolving Hex dependencies...
Dependency resolution completed:
  mime 1.2.0
  phoenix 1.3.2
  phoenix_pubsub 1.0.2
  plug 1.5.0
  poison 3.1.0
* Getting phoenix (Hex package)
* Getting phoenix_pubsub (Hex package)
* Getting plug (Hex package)
* Getting poison (Hex package)
* Getting mime (Hex package)
Unchecked dependencies for environment prod:
* cowboy (Hex package)
  the dependency is not available, run "mix deps.get"
** (Mix) Can't continue due to errors on dependencies

[NobbZ]

You can install phoenix as an archive? Shouldn't the archive be phx_new without any dependencies?

[josevalim]

@Eiji7 it makes sense. We first install phoenix a dependency which means optional dependencies are not retrieved. We may need an internal flag here although that may require changes to Hex.

[ericmj]

I agree that it probably shouldn't crash but what's the use case here? It's not clear to me if optional dependencies should be fetched and used or not fetched and ignored.

[josevalim]

@ericmj optional dependencies are always fetched on the parent but because we fetch Phoenix as a dependency, the optional deps are not fetched.

[ericmj]

Right, I understand what is happening, but what should happen?

Optional dependencies are in fact optional so maybe they should not be fetched. We can't use phoenix and ecto as examples because they were not intended as archives. Until we have a use case it's not clear what the fix should be.

[Eiji7]

@NobbZ You can install phoenix as an archive? Shouldn't the archive be phx_new without any dependencies?

Yes, but not every repository have *.ez files and sometimes there is a need to compile them. I could do it manually or just use simple task with hex name.

@ericmj We can't use phoenix and ecto as examples because they were not intended as archives.

No? https://hexdocs.pm/phoenix/installation.html#phoenix
ecto was just another example for optional packages - i.e. I want to say that's not only phoenix-related problem

@ericmj: Optional dependencies are in fact optional so maybe they should not be fetched. (…) Until we have a use case it's not clear what the fix should be.

I would like to install some packages (with mix tasks) as archives. Good examples here are credo or inch_ex. I could add them to every project or just install archives. I think that we should fetch all dependencies unless there is a good way to determine if any task requires optional dependencies.

[josevalim]

@ericmj it should behave the same as if i git cloned phoenix and ran mix archive.install. The fact we use an intermediate dependnecy is an impleentation detail and it should not "leak".

[ericmj]

No? https://hexdocs.pm/phoenix/installation.html#phoenix
ecto was just another example for optional packages - i.e. I want to say that's not only phoenix-related problem

Yes :). The phoenix installer is a separate project that has no dependencies and works with mix archive.install.

I would like to install some packages (with mix tasks) as archives. Good examples here are credo or inch_ex. I could add them to every project or just install archives. I think that we should fetch all dependencies unless there is a good way to determine if any task requires optional dependencies.

I would not recommend this. Archives will only work in special circumstances and if the project was not intended to be used as an archive it will likely not work well. Archives only use their dependencies during compilation so they will not be included in the archive itself because they would otherwise conflict with the dependencies in your projects. It's unlikely that credo and inch_ex will work correctly since they both use poison as dependency.

[ericmj]

it should behave the same as if i git cloned phoenix and ran mix archive.install. The fact we use an intermediate dependnecy is an impleentation detail and it should not "leak".

Good point 👍

[Eiji7]

@ericmj: I would not recommend this. Archives will only work in special circumstances and if the project was not intended to be used as an archive it will likely not work well.

I want to install archives just for special circumstances. Any package which task analyses project code or generates code based on current working directory should not be a problem to install as archive. At least packages which I'm manually installing are not a problem.

@ericmj Archives only use their dependencies during compilation so they will not be included in the archive itself because they would otherwise conflict with the dependencies in your projects.

And that's perfect, I don't wish for more 😃

@ericmj It's unlikely that credo and inch_ex will work correctly since they both use poison as dependency.

They works for me without any problem with just few really simple commands:

mix archive.install bunt # required by credo archive
mix archive.install credo
mix archive.install poison # required by inch archive
mix archive.install inch

However I can see problem which you tries to say. Normally I would agree, but packages which focuses only on things like code analyze and generating code does affect other projects as much as normal libraries. They are mostly or only used for tasks and there is no problem for projects to update a major version of such dependency, so they could be updated really often and if tasks works then everything is ok. This could be a problem if such package is no longer maintained or is really rarely updated.

[josevalim]

They works for me without any problem with just few really simple commands:

That's a bad idea because now poison in your project is going to conflict with the packages installed in your system. For example, you could even write a library, starting using poison, forget to add it as a dependency since you have locally installed, and then find it out only when you push your code.

Global packages are a bad idea and we should in no way be promoting those.

The reason why we need to fix this issue is not because of archives, because as @ericmj said, they are not supposed to have dependencies anyway, but because escript.install will have this same issue.

[Eiji7]

@josevalim That's a bad idea because now poison in your project is going to conflict with the packages installed in your system. For example, you could even write a library, starting using poison, forget to add it as a dependency since you have locally installed, and then find it out only when you push your code.

It's why I do not install many archives. I'm installing only those which applies to all / most of all my projects.

btw. those 4 packages I installed just now (temporary) to test if them work at same time to write a response to @ericmj

Anyway before release I'm running Ubuntu VM, copying install script and run, so I'm installing asdf, erlang, elixir and finally clone my repo and run mix do deps.get, test on clean environment on most popular distribution.

And anyway, anyway my projects are using jason (of course if needed). 😃

[josevalim]

Closing in favor of #7835.