New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Manage a global package environment for easy GHCi access #3737

Open
ezyang opened this Issue Aug 30, 2016 · 22 comments

Comments

Projects
None yet
9 participants
@ezyang
Contributor

ezyang commented Aug 30, 2016

See https://www.reddit.com/r/haskell/comments/507h0p/cabal_newbuild_is_a_package_manager/d72bzce?context=3

I think, essentially, the request is for cabal new-install to put the requested package into some global packages set, which we keep up-to-date upon invocations to new-install. Maybe we should even write the package environment in the place GHC looks by default, so a bare ghci invocation works.

@hvr thought extra-packages might be what you want (if it were implemented) but I'm not sure; I think it would cause any such mentioned packages to be considered local, and thus built inplace. But maybe that's a misinterpretation.

CC @cocreature

@cocreature

This comment has been minimized.

Show comment
Hide comment
@cocreature

cocreature Aug 30, 2016

Collaborator

Is there any documentation on extra-packages? Having them treated as local packages doesn’t sound like what I want.

Personally I like viewing the global package set as just another project that is chosen if no other project is found. That removes the need for separate commands in the UI. Thinking about it at least the name extra-packages sounds like what I want, so just a list of packages that should be available in a project without them being declared as a dependency in a cabal file. For packages cabal new-install could just add the package to that list automatically and build it. If I am in a project it gets installed there or if I am somewhere else it gets installed into the global project.

Collaborator

cocreature commented Aug 30, 2016

Is there any documentation on extra-packages? Having them treated as local packages doesn’t sound like what I want.

Personally I like viewing the global package set as just another project that is chosen if no other project is found. That removes the need for separate commands in the UI. Thinking about it at least the name extra-packages sounds like what I want, so just a list of packages that should be available in a project without them being declared as a dependency in a cabal file. For packages cabal new-install could just add the package to that list automatically and build it. If I am in a project it gets installed there or if I am somewhere else it gets installed into the global project.

@ezyang

This comment has been minimized.

Show comment
Hide comment
@ezyang

ezyang Aug 30, 2016

Contributor

There is no documentation on extra-packages; but to its credit, they are also not implemented (only the syntax bits are in.) It's hard to say what the semantics were supposed to be... @dcoutts ? I do think your interpretation makes sense.

Contributor

ezyang commented Aug 30, 2016

There is no documentation on extra-packages; but to its credit, they are also not implemented (only the syntax bits are in.) It's hard to say what the semantics were supposed to be... @dcoutts ? I do think your interpretation makes sense.

@dcoutts

This comment has been minimized.

Show comment
Hide comment
@dcoutts

dcoutts Aug 31, 2016

Member

The motivation for extra-packages is things like ghc-vis, libraries that you want to have available in your development environment to use in ghci, but that is not a dependency of any package in your project.

So extra-packages are to be added into the project environment, solved for etc. They can come from the store or be built locally following the usual rule. They're additional to the ones automatically pulled in by deps of local packages. So where this really shows up is that you can use them as a target of build (e.g. if it's not built in the store already) and that they should go into the .ghc.environment file.

If the extra-packages is a exe then I guess it means (apart from building etc) just make it available in any managed shell environment or managed bin dir we might do.

Member

dcoutts commented Aug 31, 2016

The motivation for extra-packages is things like ghc-vis, libraries that you want to have available in your development environment to use in ghci, but that is not a dependency of any package in your project.

So extra-packages are to be added into the project environment, solved for etc. They can come from the store or be built locally following the usual rule. They're additional to the ones automatically pulled in by deps of local packages. So where this really shows up is that you can use them as a target of build (e.g. if it's not built in the store already) and that they should go into the .ghc.environment file.

If the extra-packages is a exe then I guess it means (apart from building etc) just make it available in any managed shell environment or managed bin dir we might do.

@hvr

This comment has been minimized.

Show comment
Hide comment
@hvr

hvr Aug 31, 2016

Member

If the extra-packages is a exe then I guess it means (apart from building etc) just make it available in any managed shell environment or managed bin dir we might do.

For consistency with the per-components framework, I'd expect extra-packages to build only library components (although maybe it should be renamed to extra-libs), and for when exe components are to be requested there should be a separate extra-tools directive (c.f. #3708)

Member

hvr commented Aug 31, 2016

If the extra-packages is a exe then I guess it means (apart from building etc) just make it available in any managed shell environment or managed bin dir we might do.

For consistency with the per-components framework, I'd expect extra-packages to build only library components (although maybe it should be renamed to extra-libs), and for when exe components are to be requested there should be a separate extra-tools directive (c.f. #3708)

@cocreature

This comment has been minimized.

Show comment
Hide comment
@cocreature

cocreature Sep 2, 2016

Collaborator

Reading @dcoutts’ description, extra-packages in combination with a global project that all commands default to if there is no cabal.project (or cabal file) seems like it would be exactly what I’m looking for.

Collaborator

cocreature commented Sep 2, 2016

Reading @dcoutts’ description, extra-packages in combination with a global project that all commands default to if there is no cabal.project (or cabal file) seems like it would be exactly what I’m looking for.

@23Skidoo

This comment has been minimized.

Show comment
Hide comment
@23Skidoo

23Skidoo Sep 2, 2016

Member

@dcoutts Do we plan to make cabal new-install foo update extra-packages in the global project?

Member

23Skidoo commented Sep 2, 2016

@dcoutts Do we plan to make cabal new-install foo update extra-packages in the global project?

@dcoutts

This comment has been minimized.

Show comment
Hide comment
@dcoutts

dcoutts Sep 8, 2016

Member

@23Skidoo I'm not sure I'd totally nailed down what global install of libs should mean, but if we can make it behave more or less like a project then that would be easier to implement and might be a good UI as it'd follow the same mental model. So if we can say that the interpretation of install of libs user-wide is to update a virtual or real user-wide project with extra-packages and rebuild then great. But note that this implies full consistency, as with projects, and possibly rebuilding other packages that were previously installed to make them consistent (though at least we can do atomic switchovers so we don't bork things if rebuilding fails).

Member

dcoutts commented Sep 8, 2016

@23Skidoo I'm not sure I'd totally nailed down what global install of libs should mean, but if we can make it behave more or less like a project then that would be easier to implement and might be a good UI as it'd follow the same mental model. So if we can say that the interpretation of install of libs user-wide is to update a virtual or real user-wide project with extra-packages and rebuild then great. But note that this implies full consistency, as with projects, and possibly rebuilding other packages that were previously installed to make them consistent (though at least we can do atomic switchovers so we don't bork things if rebuilding fails).

@dcoutts

This comment has been minimized.

Show comment
Hide comment
@dcoutts

dcoutts Sep 8, 2016

Member

Oh and then note that this is not at all as simple as what Simon Marlow was suggesting in that same discussion which was to just blindly register the lib in the user db ("I know what I'm doing dammit!").

So I'd suggest that we proceed by implementing extra-packages for local projects and in parallel think about whether it fully works UI-wise to have the user-wide environment (ghc/ghci libs and exes) managed like a local project.

First thing to note there is that you don't want full consistency for deps of exes. I should be able to install an exe user-wide even if it doesn't have the same consistent deps as other user-wide binaries. So perhaps we should say that libraries available user-wide are managed like a project, but exes are done differently (or equivalently they're solved for independently aka --independent-goals but --independent-exe-goals only).

Member

dcoutts commented Sep 8, 2016

Oh and then note that this is not at all as simple as what Simon Marlow was suggesting in that same discussion which was to just blindly register the lib in the user db ("I know what I'm doing dammit!").

So I'd suggest that we proceed by implementing extra-packages for local projects and in parallel think about whether it fully works UI-wise to have the user-wide environment (ghc/ghci libs and exes) managed like a local project.

First thing to note there is that you don't want full consistency for deps of exes. I should be able to install an exe user-wide even if it doesn't have the same consistent deps as other user-wide binaries. So perhaps we should say that libraries available user-wide are managed like a project, but exes are done differently (or equivalently they're solved for independently aka --independent-goals but --independent-exe-goals only).

@ezyang

This comment has been minimized.

Show comment
Hide comment
@ezyang

ezyang Sep 8, 2016

Contributor

You are right that there is a tension between (1) minimizing build time (if I say cabal new-install, I want the package to install ASAP) and (2) keeping a consistent database. Old cabal install always tried to minimize build time, by trying to use preexisting packages and being willing to install inconsistent things.

You might think that if you are willing to wait, it would always be right to keep the database consistent. But this might not be possible if a user wants to add two separate packages to their environment which don't have compatible dependencies. So inconsistency can be useful, even if it is not completely right. So, we could try the "global project" model, but it definitely sounds like letting each of the library roots be an independent goal would let more plans go through. Fortunately, I don't see why extra-packages couldn't be setup to allow independent goals, so you still ought to be able to use the project codepath. But maybe you want to warn when some of the dependencies are inconsistent!

Contributor

ezyang commented Sep 8, 2016

You are right that there is a tension between (1) minimizing build time (if I say cabal new-install, I want the package to install ASAP) and (2) keeping a consistent database. Old cabal install always tried to minimize build time, by trying to use preexisting packages and being willing to install inconsistent things.

You might think that if you are willing to wait, it would always be right to keep the database consistent. But this might not be possible if a user wants to add two separate packages to their environment which don't have compatible dependencies. So inconsistency can be useful, even if it is not completely right. So, we could try the "global project" model, but it definitely sounds like letting each of the library roots be an independent goal would let more plans go through. Fortunately, I don't see why extra-packages couldn't be setup to allow independent goals, so you still ought to be able to use the project codepath. But maybe you want to warn when some of the dependencies are inconsistent!

@simonmar

This comment has been minimized.

Show comment
Hide comment
@simonmar

simonmar Oct 8, 2016

Member

I'd like to suggest something really really simple: have cabal new-install for a library build and install the library into the store.

Additionally it would be nice to have a way to do ghci -package-db ~/.cabal/store/ghc-8.0.1/package.db via cabal, but I can always do that myself.

I'm sure you're thinking "but why would you do that? You'll get a random collection of packages". My point is that usually it does the right thing, and when it doesn't do the right thing I can give it a hint by adding a -package flag. I don't want to mess with global state or a global package environment or environment files or anything - just let me build a package and use it.

Member

simonmar commented Oct 8, 2016

I'd like to suggest something really really simple: have cabal new-install for a library build and install the library into the store.

Additionally it would be nice to have a way to do ghci -package-db ~/.cabal/store/ghc-8.0.1/package.db via cabal, but I can always do that myself.

I'm sure you're thinking "but why would you do that? You'll get a random collection of packages". My point is that usually it does the right thing, and when it doesn't do the right thing I can give it a hint by adding a -package flag. I don't want to mess with global state or a global package environment or environment files or anything - just let me build a package and use it.

@hvr

This comment has been minimized.

Show comment
Hide comment
@hvr

hvr Oct 8, 2016

Member

@simonmar You don't get a random collection of packages, but rather a forrest of install-plans into scope. But your suggestion would work fine, if you care only about a single package. However, if quickly testing something in GHCI is your interest, there's a related feature #3843 which has a similiar problem. So what about having a convenient command: cabal ghci --transitive-deps <list-of-deps> which throws you into a GHCi session with those packages in scope (possibly building them first), as well as their transitive deps (but in a consistent plan)?

EDIT: Just to clarify, I think the cabal new-install <library-package> suggestion is reasonable in its own right (I currently emulate it by creating a dummy package .cabal file, which contains a build-depends with the libraries I want to get populated into the nix store); my comment is rather about the 2nd part with ghci-invocation sugar.

Member

hvr commented Oct 8, 2016

@simonmar You don't get a random collection of packages, but rather a forrest of install-plans into scope. But your suggestion would work fine, if you care only about a single package. However, if quickly testing something in GHCI is your interest, there's a related feature #3843 which has a similiar problem. So what about having a convenient command: cabal ghci --transitive-deps <list-of-deps> which throws you into a GHCi session with those packages in scope (possibly building them first), as well as their transitive deps (but in a consistent plan)?

EDIT: Just to clarify, I think the cabal new-install <library-package> suggestion is reasonable in its own right (I currently emulate it by creating a dummy package .cabal file, which contains a build-depends with the libraries I want to get populated into the nix store); my comment is rather about the 2nd part with ghci-invocation sugar.

@ezyang

This comment has been minimized.

Show comment
Hide comment
@ezyang

ezyang Oct 8, 2016

Contributor

Well, I don't see any reason why we can't go ahead and implement @simonmar's suggestion. I don't think it will be that useful, but it would be pretty easy to do.

Contributor

ezyang commented Oct 8, 2016

Well, I don't see any reason why we can't go ahead and implement @simonmar's suggestion. I don't think it will be that useful, but it would be pretty easy to do.

@phadej

This comment has been minimized.

Show comment
Hide comment
@phadej

phadej Oct 9, 2016

Collaborator

I realised on the plane: It would be useful, to tell cabal to only use packages in the store, or solver to consider only versions with tars cached. I had to reinvent constraints manually to make offline work possible :) in that light new-install will be useful to populate caches

Collaborator

phadej commented Oct 9, 2016

I realised on the plane: It would be useful, to tell cabal to only use packages in the store, or solver to consider only versions with tars cached. I had to reinvent constraints manually to make offline work possible :) in that light new-install will be useful to populate caches

@23Skidoo

This comment has been minimized.

Show comment
Hide comment
@23Skidoo

23Skidoo Oct 9, 2016

Member

@phadej Do you know that cabal install has an --offline flag?

Member

23Skidoo commented Oct 9, 2016

@phadej Do you know that cabal install has an --offline flag?

@phadej

This comment has been minimized.

Show comment
Hide comment
@phadej

phadej Oct 9, 2016

Collaborator

@23Skidoo but it doesn't work with new-build :(

Collaborator

phadej commented Oct 9, 2016

@23Skidoo but it doesn't work with new-build :(

@ezyang

This comment has been minimized.

Show comment
Hide comment
@ezyang

ezyang Oct 10, 2016

Contributor

Yes, offline mode would be great. But it does make the process less deterministic (because the solver has to prefer versions for which we have local tarballs.)

Contributor

ezyang commented Oct 10, 2016

Yes, offline mode would be great. But it does make the process less deterministic (because the solver has to prefer versions for which we have local tarballs.)

@tom-bop

This comment has been minimized.

Show comment
Hide comment
@tom-bop

tom-bop Oct 25, 2017

For people who don't think this feature will be useful, can I clarify?: it's my understanding that, without using "old-install", the ghci command becomes effectively useless, except to toy with functions from the Prelude. Is that right?

tom-bop commented Oct 25, 2017

For people who don't think this feature will be useful, can I clarify?: it's my understanding that, without using "old-install", the ghci command becomes effectively useless, except to toy with functions from the Prelude. Is that right?

@fgaz

This comment has been minimized.

Show comment
Hide comment
@fgaz

fgaz Oct 27, 2017

Collaborator

@tom-bop
Well, except that now there are ghc environment files. So if you invoke ghci in a project directory it will include the libraries your packages depend on.

new-install for libraries will basically be a global env file plus #4825 plus some a lot of consistency checking

Collaborator

fgaz commented Oct 27, 2017

@tom-bop
Well, except that now there are ghc environment files. So if you invoke ghci in a project directory it will include the libraries your packages depend on.

new-install for libraries will basically be a global env file plus #4825 plus some a lot of consistency checking

@fgaz

This comment has been minimized.

Show comment
Hide comment
@fgaz

fgaz Oct 27, 2017

Collaborator

@simonmar #4825 does what you want

Collaborator

fgaz commented Oct 27, 2017

@simonmar #4825 does what you want

@tom-bop

This comment has been minimized.

Show comment
Hide comment
@tom-bop

tom-bop Oct 27, 2017

@fgaz #4825 is awesome, but is for nonlocal executables, and @simonmar and I are talking about nonlocal libraries. (Unless you're saying that #4825 additionally installs packages in the global store?)

tom-bop commented Oct 27, 2017

@fgaz #4825 is awesome, but is for nonlocal executables, and @simonmar and I are talking about nonlocal libraries. (Unless you're saying that #4825 additionally installs packages in the global store?)

@fgaz

This comment has been minimized.

Show comment
Hide comment
@fgaz

fgaz Oct 27, 2017

Collaborator

@tom-bop Yeah, it does. It installs the selected package in the store and symlinks its exes, if there are any. So you can use it to add libraries to the store too. You'll have to manually create a ghc environment file or explicitly pass -package* flags to ghci though.

Collaborator

fgaz commented Oct 27, 2017

@tom-bop Yeah, it does. It installs the selected package in the store and symlinks its exes, if there are any. So you can use it to add libraries to the store too. You'll have to manually create a ghc environment file or explicitly pass -package* flags to ghci though.

@hvr

This comment has been minimized.

Show comment
Hide comment
@hvr

hvr Oct 27, 2017

Member

@tom-bop @fgaz @simonmar you may want to take a look at https://gist.github.com/a076fbd34083a44faecd45ffd12b9772 which is a shell script I've been using to create package environment files (which exposes all transitive deps for the dependency spec specified at creation). It requires cabal 2.1 and GHC 8.0.2 (or later).

Member

hvr commented Oct 27, 2017

@tom-bop @fgaz @simonmar you may want to take a look at https://gist.github.com/a076fbd34083a44faecd45ffd12b9772 which is a shell script I've been using to create package environment files (which exposes all transitive deps for the dependency spec specified at creation). It requires cabal 2.1 and GHC 8.0.2 (or later).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment