Question: Why 'Flat' Dependencies

[iamdoron]

Hi,

I'm new to elm. I wanted to know why elm & elm-package were designed in a way, where all packages are installed locally (which is good) but in a flat structure, and of course - are there any plans on evolving it?.
For example,

• johnpmayer/elm-webgl 1.0.1 depends on johnpmayer/elm-linear-algebra 2.x
• elm-lang.org stable depends on johnpmayer/elm-webgl 1.x, johnpmayer/elm-linear-algebra 1.x

Try to elm-package install elm-lang.org stable and you'll get:

Error: Unable to find a set of packages that will work with your constraints.

I guess that from the compiler point-of-view it's much easier to implement this mechanism, but when the 'eco-system' will grow it will happen much frequently, and it will be unpleasant. I have to say it worries me, as someone who is considering using elm-lang. I'm coming from node, where npm really does a good job in managing dependencies.

As I said, I'm new to elm, so there's a chance I misunderstood how packages work 😄

[johnpmayer]

Elm is statically typed, so we'd need to introduce a construct to know that
the type Foo from package Bar 1.0 is the same type and has the same
internal representation as the type Foo from package Bar 2.0.
On Jan 11, 2015 8:10 AM, "iamdoron" notifications@github.com wrote:

Hi,

I'm new to elm. I wanted to know why elm & elm-package were designed in a
way, where all packages are installed locally (which is good) but in a flat
structure, and of course - are there any plans on evolving it?.
For example,

• johnpmayer/elm-webgl 1.0.1 depends on johnpmayer/elm-linear-algebra
  2.x
• elm-lang.org stable depends on johnpmayer/elm-webgl 1.x, johnpmayer/elm-linear-algebra
  1.x

Try to elm-package install elm-lang.org stable and you'll get:

Error: Unable to find a set of packages that will work with your
constraints.

I guess that from the compiler point-of-view it's much easier to implement
this mechanism, but when the 'eco-system' will grow it will happen much
frequently, and it will be unpleasant. I have to say it worries me, as
someone who is considering using elm-lang. I'm coming from node, where npm
https://www.npmjs.com really does a good job in managing dependencies.

As I said, I'm new to elm, so there's a chance I misunderstood how
packages work [image: 😄]

—
Reply to this email directly or view it on GitHub
#89.

[iamdoron]

It gets event more complicated when somewhere in the api of a package there's a function that returns a type from a different package - I'm not aware of a good solution for it (there is this notion of peer dependency, where you specify a restriction of a companion/sibling package)

[evancz]

This choice was made intentionally. I have researched this quite a lot and weighed the tradeoffs for quite a while. Unfortunately, I have not done a big writeup about all of this yet!

Imagine that version 1.0.0 of List implements the data structure in a certain way and version 1.0.1 implements them in a different way. From the user's perspective, nothing has changed. The API is the same. In generated code, a function that takes 1.0.0 lists will fail when it is given a 1.0.1 list. The same problem is true in JS, but in that world, there is not great diversity of data structures and people are generally not surprised when there is a runtime error.

One way to deal with this is to treat the list types differently, so List-1.0.0 is not the same as List-1.0.1 in the type system. That works for a while, but it introduces a much much crazier kind of issue. Say you are an industrial user and you are 3 years into a project, and finally two lists from totally different corners of the project meet. It can't work! But by this time, so much code has been written that the cost of fixing it is immense. I believe this problem can arise in Java. This can also lead to a level of fragmentation in your libraries that is worse than the original problem we were trying to solve. Another thing to think about is code size. If there are 10 versions of the List library in every Elm program, this is pretty silly.

I personally think the question should be "how can we make things pleasant?" The tradeoffs in JS are not the same ones faced here. The longer term plan for elm-package is to be bumping version bounds based on automated testing, such that maintainers are not a big bottleneck on keeping version bounds as wide as possible.

[iamdoron]

I'm not sure I understand. If there are 10 versions of List, why not just take the highest one? (if it's stable and hadn't change much - if there was a major change you have a problem either way)

Essentially, what you are saying, let's restrict it from the beginning - all your packages that use List must use the same version or your project will not compile (which is a really good restriction), but (prepare for success) when many packages will be built, you have to make them use the same version you do (and the tree can get complex). If the major version doesn't change - no problem, but if it does, it's an issue. That's why I think we should differentiate between inner dependencies to ones you give in your package API. It's good restriction for an 'outer' dependency, but for an 'inner' one, does it matter?

Another point, is that the error is weird (Unable to find a set of packages that will work with your constraints.).

[iamdoron]

[more about the error]
Once I know there's an error, what should I do now? should I PR to what package? how do I solve it?
And even more - does it a major change when you change a dependency? It might, like the case with the packages in my original question

[evancz]

Is this a question about the specific constraint thing with @johnpmayer's stuff? Can 1.0.1 not work with less than 2.0.0 of the other? It's fine to change the constraints in the elm-lang.org repo.

[iamdoron]

I'm not talking about a specific issue, it's just an example I faced the first day trying to use elm-package - of course it doesn't mean you see it everywhere.
So, to sum up, my concerns are

• the flat packages design creates an invisible dependency among all the packages in a 'dependency tree'. We can divide these 'invisible' dependencies to 'inner' & 'outer' dependencies - packages that are not part of a package interface and packages that are.
• the error message doesn't guide us to solve the problem - for a big 'dependency tree' it could get messy
• this 'invisible' dependency might result in a major version change - ideally only for 'outer' dependencies, right now it can happen also for 'inner', and the automatic tool (which I learned exist only today) doesn't take it into account

I think that's it 😄

[evancz]

Yep, point 1 is a known technique that we have considered. We can determine public/private dependencies automatically based on API to a point. The only problem is that values can leak through that do not appear in types, which we have no way to detect reliably. This may be particular constants or anonymous functions. It is not clear how big of a problem that is, so I would want to figure out the worst possible tricks we could do. Perhaps nothing catastrophic like what I described in my earlier comment can happen? I have thought it through well enough to say that with confidence though, and the policy in Elm in general is "adding freedom is easy, taking it away is hard" so err on the side of caution when you are not sure about something that has big implications.

Overall, you should know that this is a young project. This is essentially the first draft of a lot of things, and there are a ton of plans of how things will develop, mostly around more automated testing and automatically expanding version bounds. To make it clearer how new this is: this is the first version that actually does any dependency solving at all.

I think points 2 and 3 in your latest comment deserve issues to track progress on this if they can be made specific. In particular, dependency solving is relatively rudimentary right now, so that may be relatively easy to improve error messages. That is vital to creating the best elm-package and a priority for me, so please share if you have specific ideas of how to make that happen or want to contribute code.

For point 3, you can think of the current "dependencies" listing as "public-dependencies" and there just are no private dependencies right now. It seems plausible to incorporate changes there into the version bump calculation, but I am not sure what the rules should be really. It'd be good to have a set of cases where a dependency can introduce major or minor changes and see to what extent that can be detected statically.

[xixixao]

@evancz Could you elaborate on how values of certain types could leak through without being captured by the static types of the API? With an example perhaps?

[seanstrom]

@evancz have you looked into how Rust lang handles dependencies?
They seem to have similar problems but do permit nested dependencies.

[Risto-Stevcev]

@evancz Is it possible to remove partial functions altogether? Then you could actually determine public/private dependencies and that would solve the problem in the best way.

It seems as though whatever can be expressed with partial functions could be expressed in other ways. The obvious one:

Haskell:

Prelude> :type head
head :: [a] -> a
Prelude> head []
*** Exception: Prelude.head: empty list

Elm:

head : List a -> Maybe a
head [] == Nothing