Finish new-install

[typedrat]

Closes #4558.

Adds support for:

• Local targets
• Libraries

--

Please include the following checklist in your PR:

• Patches conform to the coding conventions.
• Any changes that could be relevant to users have been recorded in the changelog.
• The documentation has been updated, if necessary.
• If the change is docs-only, [ci skip] is used to avoid triggering the build bots.

[typedrat]

Okay, so this is now feature-complete in the loosest sense.

The following cases still need to be handled:

• Tell the user when environment files aren't supported (and therefore libraries can't be installed)
• Quit out early when it will be a no-op.
• Make the local package requirements not included when they aren't relevant
• Fix bugs
• Document

[typedrat]

Okay, so this is now feature-complete in the loosest sense.

The following cases still need to be handled:

• Tell the user when environment files aren't supported (and therefore libraries can't be installed)
• Quit out early when it will be a no-op.
• Make the local package requirements not included when they aren't relevant
• Fix bugs
• Document

[typedrat]

❯ cat ~/.ghc/x86_64-darwin-8.4.2/environments/default
package-id lnr-1.20.7-359f8114

That's what I get after I try to install linear. Why do I not get the vowels?

[fgaz]

Are you on mac os?

https://github.com/typedrat/cabal/blob/638c78e790721566780dbf75a2e6f884ceafa970/cabal-install/Distribution/Client/PackageHash.hs#L164

[typedrat]

Hmm, how would I get around that...

[alexbiehl]

Awww I fucked up the merge with the Github online editor! Sorry about that!

[alexbiehl]

Awww I fucked up the merge with the Github online editor! Sorry about that!

[23Skidoo]

Just rebase.

[23Skidoo]

Just rebase.

[typedrat]

Should be done, at least done enough to get smacked around by other people. It works for me™ but I haven't had time/access to machines to test it on anything but macOS.

[typedrat]

Should be done, at least done enough to get smacked around by other people. It works for me™ but I haven't had time/access to machines to test it on anything but macOS.

[typedrat]

7.4 is legitimately broken, but it's also outside of our support window.

[typedrat]

7.4 is legitimately broken, but it's also outside of our support window.

[quasicomputational]

It's not working for me:

$ cabal new-run cabal -- -v2 new-install lib:Cabal
Resolving dependencies...
Up to date
Reading available packages of hackage.haskell.org...
Using most recent state specified from most recent cabal update
index-state(hackage.haskell.org) = 2018-07-02T15:20:16Z
Wrote tarball sdist to
/home/sam2/src/cabal/dist-newstyle/sdist/Cabal-2.3.0.0.tar.gz
Wrote tarball sdist to
/home/sam2/src/cabal/dist-newstyle/sdist/cabal-testsuite-2.3.0.0.tar.gz
Wrote tarball sdist to
/home/sam2/src/cabal/dist-newstyle/sdist/cabal-install-2.3.0.0.tar.gz
Wrote tarball sdist to
/home/sam2/src/cabal/dist-newstyle/sdist/solver-benchmarks-2.3.0.0.tar.gz
Wrote tarball sdist to
/home/sam2/src/cabal/dist-newstyle/sdist/pretty-show-1.6.16.tar.gz
Resolving dependencies...
/home/sam2/.local/bin/ghc --numeric-version
looking for tool ghc-pkg near compiler in /home/sam2/.local/bin
found ghc-pkg in /home/sam2/.local/bin/ghc-pkg
/home/sam2/.local/bin/ghc-pkg --version
/home/sam2/.local/bin/ghc --supported-languages
/home/sam2/.local/bin/ghc --info
Distribution/Simple/GHC.hs:453:5-53: Irrefutable pattern failed for pattern Just ghcProg

That GHC it's trying to use is 8.4.2.

[quasicomputational]

It's not working for me:

$ cabal new-run cabal -- -v2 new-install lib:Cabal
Resolving dependencies...
Up to date
Reading available packages of hackage.haskell.org...
Using most recent state specified from most recent cabal update
index-state(hackage.haskell.org) = 2018-07-02T15:20:16Z
Wrote tarball sdist to
/home/sam2/src/cabal/dist-newstyle/sdist/Cabal-2.3.0.0.tar.gz
Wrote tarball sdist to
/home/sam2/src/cabal/dist-newstyle/sdist/cabal-testsuite-2.3.0.0.tar.gz
Wrote tarball sdist to
/home/sam2/src/cabal/dist-newstyle/sdist/cabal-install-2.3.0.0.tar.gz
Wrote tarball sdist to
/home/sam2/src/cabal/dist-newstyle/sdist/solver-benchmarks-2.3.0.0.tar.gz
Wrote tarball sdist to
/home/sam2/src/cabal/dist-newstyle/sdist/pretty-show-1.6.16.tar.gz
Resolving dependencies...
/home/sam2/.local/bin/ghc --numeric-version
looking for tool ghc-pkg near compiler in /home/sam2/.local/bin
found ghc-pkg in /home/sam2/.local/bin/ghc-pkg
/home/sam2/.local/bin/ghc-pkg --version
/home/sam2/.local/bin/ghc --supported-languages
/home/sam2/.local/bin/ghc --info
Distribution/Simple/GHC.hs:453:5-53: Irrefutable pattern failed for pattern Just ghcProg

That GHC it's trying to use is 8.4.2.

[quasicomputational]

Haven't been able to actually try the command out yet, but here are some notes from looking at the source.

[quasicomputational]

GHC has code for parsing these files in DynFlags, but this code is better and we should have an eye on reusing it in GHC. Specifically, I think exporting parseEnvironmentFile along with GhcEnvironmentFileEntry and its constructors from a non-internal module ought to be enough (which we ought to do anyway, since it's mentioned in the types of this non-internal module).

[typedrat]

Already exposed in Distribution.Simple.GHC. When we get this merged and available to GHC, I'd be happy to write a patch to use this instead. :)

[quasicomputational]

In GitHub's infinite wisdom I can't attach review comments to lines that aren't changed, but there's a comment above with some TODOs. Those are solved now, right?

[typedrat]

Yep.

[typedrat]

Should have fixed your issue, @quasicomputational.

[typedrat]

Should have fixed your issue, @quasicomputational.

[typedrat]

Why is there a parse error only on 7.10?

[typedrat]

Why is there a parse error only on 7.10?

[quasicomputational]

Sadly not, still happening with 0b2817f (this PR's latest commit).

[quasicomputational]

Sadly not, still happening with 0b2817f (this PR's latest commit).

[typedrat]

For some reason, it doesn't have your compiler in the program db it passes to getInstalledPackages.

[typedrat]

For some reason, it doesn't have your compiler in the program db it passes to getInstalledPackages.

[quasicomputational]

No clue here, sorry. I can provide diagnostics if you've got any idea what might be helpful, or any new logging you put in to try and work out what's going on.

[quasicomputational]

No clue here, sorry. I can provide diagnostics if you've got any idea what might be helpful, or any new logging you put in to try and work out what's going on.

[typedrat]

I'll work on it more seriously after breakfast. 😝

[typedrat]

I'll work on it more seriously after breakfast. 😝

[typedrat]

    print progDb

on line CmdInstall.hs:319 would be helpful.

[typedrat]

    print progDb

on line CmdInstall.hs:319 would be helpful.

[quasicomputational]

$ cabal new-run cabal -- new-install -v2 lib:Cabal
Resolving dependencies...
Up to date
Reading available packages of hackage.haskell.org...
Using most recent state specified from most recent cabal update
index-state(hackage.haskell.org) = 2018-07-07T12:53:49Z
Wrote tarball sdist to
/home/sam2/src/cabal/dist-newstyle/sdist/Cabal-2.3.0.0.tar.gz
Wrote tarball sdist to
/home/sam2/src/cabal/dist-newstyle/sdist/cabal-testsuite-2.3.0.0.tar.gz
Wrote tarball sdist to
/home/sam2/src/cabal/dist-newstyle/sdist/cabal-install-2.3.0.0.tar.gz
Wrote tarball sdist to
/home/sam2/src/cabal/dist-newstyle/sdist/solver-benchmarks-2.3.0.0.tar.gz
Wrote tarball sdist to
/home/sam2/src/cabal/dist-newstyle/sdist/pretty-show-1.6.16.tar.gz
Resolving dependencies...
[]
/home/sam2/.local/bin/ghc --numeric-version
looking for tool ghc-pkg near compiler in /home/sam2/.local/bin
found ghc-pkg in /home/sam2/.local/bin/ghc-pkg
/home/sam2/.local/bin/ghc-pkg --version
/home/sam2/.local/bin/ghc --supported-languages
/home/sam2/.local/bin/ghc --info
Distribution/Simple/GHC.hs:453:5-53: Irrefutable pattern failed for pattern Just ghcProg

[quasicomputational]

$ cabal new-run cabal -- new-install -v2 lib:Cabal
Resolving dependencies...
Up to date
Reading available packages of hackage.haskell.org...
Using most recent state specified from most recent cabal update
index-state(hackage.haskell.org) = 2018-07-07T12:53:49Z
Wrote tarball sdist to
/home/sam2/src/cabal/dist-newstyle/sdist/Cabal-2.3.0.0.tar.gz
Wrote tarball sdist to
/home/sam2/src/cabal/dist-newstyle/sdist/cabal-testsuite-2.3.0.0.tar.gz
Wrote tarball sdist to
/home/sam2/src/cabal/dist-newstyle/sdist/cabal-install-2.3.0.0.tar.gz
Wrote tarball sdist to
/home/sam2/src/cabal/dist-newstyle/sdist/solver-benchmarks-2.3.0.0.tar.gz
Wrote tarball sdist to
/home/sam2/src/cabal/dist-newstyle/sdist/pretty-show-1.6.16.tar.gz
Resolving dependencies...
[]
/home/sam2/.local/bin/ghc --numeric-version
looking for tool ghc-pkg near compiler in /home/sam2/.local/bin
found ghc-pkg in /home/sam2/.local/bin/ghc-pkg
/home/sam2/.local/bin/ghc-pkg --version
/home/sam2/.local/bin/ghc --supported-languages
/home/sam2/.local/bin/ghc --info
Distribution/Simple/GHC.hs:453:5-53: Irrefutable pattern failed for pattern Just ghcProg

[quasicomputational]

Another minor docs thing: new-install is mentioned as unimplemented under "Where are my build products?", but that's not true any more! We need another example there.

[quasicomputational]

Another minor docs thing: new-install is mentioned as unimplemented under "Where are my build products?", but that's not true any more! We need another example there.

[typedrat]

@quasicomputational, you should be able to install package-ids as goals, that just slipped my mind.

[typedrat]

@quasicomputational, you should be able to install package-ids as goals, that just slipped my mind.

[typedrat]

@tom-bop, it's the sum of existing libraries (new-install a followed by new-install b gives you most of the bundled libraries, a, and b)

[typedrat]

@tom-bop, it's the sum of existing libraries (new-install a followed by new-install b gives you most of the bundled libraries, a, and b)

[tom-bop]

@typedrat 👍 , but it won't for example have any of the package you "old-installed", right?

[tom-bop]

@typedrat 👍 , but it won't for example have any of the package you "old-installed", right?

[typedrat]

@tom-bop, that's correct.

[typedrat]

@tom-bop, that's correct.

[gbaz]

I've been very tempted to introduce a different sub-command altogether for managing package environments, as the install name is a bit misleading at this point, as w/ a nix-style store, the term "installing" is not the proper mental model to have anymore IMO.

This is a good question. Whether or not we do it, it would be good to bikeshed what the "right" name might be, for conceptual clarity if nothing else. I was thinking "provide"?

[gbaz]

I've been very tempted to introduce a different sub-command altogether for managing package environments, as the install name is a bit misleading at this point, as w/ a nix-style store, the term "installing" is not the proper mental model to have anymore IMO.

This is a good question. Whether or not we do it, it would be good to bikeshed what the "right" name might be, for conceptual clarity if nothing else. I was thinking "provide"?

[fgaz]

Or env, like nix

[fgaz]

Or env, like nix

[andreabedini]

I landed here wondering why cabal new-install brittany fails (see conversation linked above and #5427).

While I am sure I don't fully understand the discussion, there's one important point I haven't seen mentioned. So here's my drive-by comment.

even when 3.0 lands, there will still be heaps of references on the internet to cabal install. Please don't break those! Beginners learn from Google and Stack Overflow, not from GitHub issues (like me :P). I would guess (but I have no data) that most of the references to cabal install are used in the context of installing a executable.

I've been very tempted to introduce a different sub-command altogether for managing package environments, as the install name is a bit misleading at this point, as w/ a nix-style store, the term "installing" is not the proper mental model to have anymore IMO.

I'd say go for it then. cabal install can emit a warning mentioning the new command.

Thanks.

[andreabedini]

I landed here wondering why cabal new-install brittany fails (see conversation linked above and #5427).

While I am sure I don't fully understand the discussion, there's one important point I haven't seen mentioned. So here's my drive-by comment.

even when 3.0 lands, there will still be heaps of references on the internet to cabal install. Please don't break those! Beginners learn from Google and Stack Overflow, not from GitHub issues (like me :P). I would guess (but I have no data) that most of the references to cabal install are used in the context of installing a executable.

I've been very tempted to introduce a different sub-command altogether for managing package environments, as the install name is a bit misleading at this point, as w/ a nix-style store, the term "installing" is not the proper mental model to have anymore IMO.

I'd say go for it then. cabal install can emit a warning mentioning the new command.

Thanks.

[ekmett]

If we're going to add v1-build, etc. we should add v2-build, etc. at the same time referencing the current new- behavior, so as to preemptively give people a forward-compat-friendly way to refer to the new build system. This would allow users who have lots of build scripts to skip one migration step, easing the transition, and ensuring that any similar changes in the future have a simple, clean, already-in-place migration plan.

[ekmett]

If we're going to add v1-build, etc. we should add v2-build, etc. at the same time referencing the current new- behavior, so as to preemptively give people a forward-compat-friendly way to refer to the new build system. This would allow users who have lots of build scripts to skip one migration step, easing the transition, and ensuring that any similar changes in the future have a simple, clean, already-in-place migration plan.

[typedrat]

@ekmett, it's less trivial than it seems to add aliases, and I feel like the likelihood of a third new paradigm is low enough to not merit the work to add them for new- commands.

[typedrat]

@ekmett, it's less trivial than it seems to add aliases, and I feel like the likelihood of a third new paradigm is low enough to not merit the work to add them for new- commands.

[ekmett]

Then I don't particularly look forward to enduring all the same complaints when we get around to a v3. ;)

[ekmett]

Then I don't particularly look forward to enduring all the same complaints when we get around to a v3. ;)

[typedrat]

Y'know what, it's not as bad as I thought, but that'd be a different PR. :)

[typedrat]

Y'know what, it's not as bad as I thought, but that'd be a different PR. :)

[hvr]

@ekmett Well, my idea was rather to consider new- as such a v2- prefix, as that's what all the scripting out there has been using for 2-3 years (it's been in cabal-1.24, cabal-2.0 and cabal-2.2); introducing an additional v2- prefix alias now seems uneconomical to me. And in the hopefully unlikely event we ever need another UI generation, we'd just call it v3- (i.e. we do not call it new- as new- shall keep referring to the 2nd gen). IOW, the prefixes would be

• v1-* refer to the 1st generation (supported since (yet unreleased) cabal-2.2.1.0)
• new-* refer to the 2nd generation (supported since cabal-1.24)
• v3-* for a future 3rd generation

[hvr]

@ekmett Well, my idea was rather to consider new- as such a v2- prefix, as that's what all the scripting out there has been using for 2-3 years (it's been in cabal-1.24, cabal-2.0 and cabal-2.2); introducing an additional v2- prefix alias now seems uneconomical to me. And in the hopefully unlikely event we ever need another UI generation, we'd just call it v3- (i.e. we do not call it new- as new- shall keep referring to the 2nd gen). IOW, the prefixes would be

• v1-* refer to the 1st generation (supported since (yet unreleased) cabal-2.2.1.0)
• new-* refer to the 2nd generation (supported since cabal-1.24)
• v3-* for a future 3rd generation

[ekmett]

@hvr You have to admit that state of affairs seems messy and inconsistent, especially in the event we ever do ship a v3-, no matter how its usage evolves.

At some unknown future point we may or may not get around to a v3-, and it may or may not want to take the new- monicker as it rolls out.

But in any of those worlds, having v2- would give a way to get current new-build style behavior in a stable way that will last through any v3- attempt, no matter if it chooses to take over new- or not. And in avoids the pretty awful inconsistency of new- referring to rather old behavior and of having every version but v2- be consistently named.

What I'd ideally want is to be able to make one change to my build scripts to migrate to v2- behavior after the next cabal release, and not to have to stir again until long after some future v3- comes along and I decide to move. While your v1-, new-, v3- story delivers on that, it does so in a way that ensures that new- is forever tainted to mean the current behavior rather than considering that future maintainers of cabal might prefer for it to switch to rolling forward, and requires folks to eventually know that new- really means old-but-not-that-old-.

[ekmett]

@hvr You have to admit that state of affairs seems messy and inconsistent, especially in the event we ever do ship a v3-, no matter how its usage evolves.

At some unknown future point we may or may not get around to a v3-, and it may or may not want to take the new- monicker as it rolls out.

But in any of those worlds, having v2- would give a way to get current new-build style behavior in a stable way that will last through any v3- attempt, no matter if it chooses to take over new- or not. And in avoids the pretty awful inconsistency of new- referring to rather old behavior and of having every version but v2- be consistently named.

What I'd ideally want is to be able to make one change to my build scripts to migrate to v2- behavior after the next cabal release, and not to have to stir again until long after some future v3- comes along and I decide to move. While your v1-, new-, v3- story delivers on that, it does so in a way that ensures that new- is forever tainted to mean the current behavior rather than considering that future maintainers of cabal might prefer for it to switch to rolling forward, and requires folks to eventually know that new- really means old-but-not-that-old-.

[hvr]

@ekmett yes, in hindsight, having used new-* at all was probably not ideal. We should have used v2- right-away; but now we're faced with the choice whether to have new-* become a dynamic prefix which will very likely break tons of scripting out there.

Also, if we decide to introduce v2-, we have a different kind of inconsistency in the event that a 3rd gen UI becomes available as tech-preview in a hypothethical future cabal-3.12 version.

• for cabal-1.24 through cabal-3.10, new-* refers to the 2nd gen UI
• for cabal-3.12 and later, new-* changes its meaning and refers to to the 3rd gen UI
• starting with cabal-2.2.1 (NB: cabal-2.2.1 may never end up in a Haskell Platform release if GHC 8.4.4 is never released) v1- and v2- become available, and refer to the 1st and 2nd gen UI
• starting cabal-3.12 and later, v3-* refers to to the 3rd gen UI

So the trade-off here is, either

• With the scheme above which changes the meaning of new-*, you'd be forced to change all your new-build-based scripts that were written over the last 2-3 years to remain well-defined (NB: v2- didn't exist yet in cabal-1.24, cabal-2.0 and cabal-2.2; so such scripting would have to require cabal-2.2.1 or later just for the sake of using the v2- prefix). So you'd suffer breakage for the benefit of avoiding an admittedly unfortunate naming weirdness or

• with the pragmatic scheme outlined in #5399 (comment), We stomach the naming weirdness, but in return we don't force all the scripting out there based on the new-* prefix to rename to v2-

In hindsight, we should have introduced v1- and v2- prefix synonyms (pointing to build and new-build respectively) back w/ cabal-1.24; but unfortunately we didn't. So now we're faced with a choice between two non-optimal tradeoffs. While I'm still leaning towards my suggestion, I'm not married to it.

[hvr]

@ekmett yes, in hindsight, having used new-* at all was probably not ideal. We should have used v2- right-away; but now we're faced with the choice whether to have new-* become a dynamic prefix which will very likely break tons of scripting out there.

Also, if we decide to introduce v2-, we have a different kind of inconsistency in the event that a 3rd gen UI becomes available as tech-preview in a hypothethical future cabal-3.12 version.

• for cabal-1.24 through cabal-3.10, new-* refers to the 2nd gen UI
• for cabal-3.12 and later, new-* changes its meaning and refers to to the 3rd gen UI
• starting with cabal-2.2.1 (NB: cabal-2.2.1 may never end up in a Haskell Platform release if GHC 8.4.4 is never released) v1- and v2- become available, and refer to the 1st and 2nd gen UI
• starting cabal-3.12 and later, v3-* refers to to the 3rd gen UI

So the trade-off here is, either

• With the scheme above which changes the meaning of new-*, you'd be forced to change all your new-build-based scripts that were written over the last 2-3 years to remain well-defined (NB: v2- didn't exist yet in cabal-1.24, cabal-2.0 and cabal-2.2; so such scripting would have to require cabal-2.2.1 or later just for the sake of using the v2- prefix). So you'd suffer breakage for the benefit of avoiding an admittedly unfortunate naming weirdness or

• with the pragmatic scheme outlined in #5399 (comment), We stomach the naming weirdness, but in return we don't force all the scripting out there based on the new-* prefix to rename to v2-

In hindsight, we should have introduced v1- and v2- prefix synonyms (pointing to build and new-build respectively) back w/ cabal-1.24; but unfortunately we didn't. So now we're faced with a choice between two non-optimal tradeoffs. While I'm still leaning towards my suggestion, I'm not married to it.

[tom-bop]

Is there a cost to simply having a v2-* alias that points to (what's currently called) new-*?

That way we'd have

• v1-*
• v2-* <-- new-* is an alias to this but over the years that could be deprecated and the new-* alias could gradually be freed up for future use (or not!)
• v3-*

[tom-bop]

Is there a cost to simply having a v2-* alias that points to (what's currently called) new-*?

That way we'd have

• v1-*
• v2-* <-- new-* is an alias to this but over the years that could be deprecated and the new-* alias could gradually be freed up for future use (or not!)
• v3-*