New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Construct a sandbox from another package repository #1196

Open
mightybyte opened this Issue Feb 13, 2013 · 16 comments

Comments

@mightybyte
Collaborator

mightybyte commented Feb 13, 2013

I routinely work on multiple projects that depend on each other but that I don't want to share a sandbox. It would be a huge time saver if there was a way to do "cabal sandbox-install" but have it get already installed packages from the old cabal --user or --global repositories (or maybe even other sandboxes) without having to rebuild them. In my use case, I would want the package to be copied to the new repository so that this sandbox still works independent of the other repository.

@tibbe

This comment has been minimized.

Show comment
Hide comment
@tibbe

tibbe Feb 13, 2013

Member

The sandbox-add-source <path> command (which will most likely be cabal sandbox add-source <path> in the release) is meant to let you depend on unreleased packages that you have checked-out in e.g. $HOME/src. add-source creates a link such that the dependency gets rebuilt if it changes. We also plan to add a --snapshot flag which would install a copy of the dependency (just like cabal-dev's add-source does).

We're generally trying to move away from installing things. Projects should be self-contained and build all their dependencies in a way that they don't clobber other projects (which is what shared, mutable package stores result in).

Member

tibbe commented Feb 13, 2013

The sandbox-add-source <path> command (which will most likely be cabal sandbox add-source <path> in the release) is meant to let you depend on unreleased packages that you have checked-out in e.g. $HOME/src. add-source creates a link such that the dependency gets rebuilt if it changes. We also plan to add a --snapshot flag which would install a copy of the dependency (just like cabal-dev's add-source does).

We're generally trying to move away from installing things. Projects should be self-contained and build all their dependencies in a way that they don't clobber other projects (which is what shared, mutable package stores result in).

@mightybyte

This comment has been minimized.

Show comment
Hide comment
@mightybyte

mightybyte Feb 13, 2013

Collaborator

I'm using add-source. But here I'm talking about packages coming from hackage, not add-source.

Collaborator

mightybyte commented Feb 13, 2013

I'm using add-source. But here I'm talking about packages coming from hackage, not add-source.

@tibbe

This comment has been minimized.

Show comment
Hide comment
@tibbe

tibbe Feb 13, 2013

Member

I misunderstood. You want the build times to be lower. We will have to wait for a Nix-style package store for that to happen. In the meantime, add jobs: $ncpus to your $HOME/.cabal/config file.

Member

tibbe commented Feb 13, 2013

I misunderstood. You want the build times to be lower. We will have to wait for a Nix-style package store for that to happen. In the meantime, add jobs: $ncpus to your $HOME/.cabal/config file.

@mightybyte

This comment has been minimized.

Show comment
Hide comment
@mightybyte

mightybyte Feb 13, 2013

Collaborator

Yeah, I've done that. Build times are still long. I guess I may not appreciate all the details, but it doesn't seem too difficult to add a flag that would include that packages from another repository in the list of things it thinks it already has.

Collaborator

mightybyte commented Feb 13, 2013

Yeah, I've done that. Build times are still long. I guess I may not appreciate all the details, but it doesn't seem too difficult to add a flag that would include that packages from another repository in the list of things it thinks it already has.

@tibbe

This comment has been minimized.

Show comment
Hide comment
@tibbe

tibbe Feb 13, 2013

Member

Not much we can do at this point I'm afraid. We cannot safely just copy installed packages around (that's why we need a Nix-style store that correctly tracks inputs so outputs can be used). Between that and making GHC faster there's not much else.

Member

tibbe commented Feb 13, 2013

Not much we can do at this point I'm afraid. We cannot safely just copy installed packages around (that's why we need a Nix-style store that correctly tracks inputs so outputs can be used). Between that and making GHC faster there's not much else.

@mightybyte

This comment has been minimized.

Show comment
Hide comment
@mightybyte

mightybyte Feb 13, 2013

Collaborator

Bummer. The case I'm talking about is really just when I'm creating a new sandbox. I've got this tree of a bunch of the dependencies I need, and they're already built. It just seems a shame to not be able to use them. Frankly, this is why I've always shied away from build sandboxes in the past--rebuilding a big stack like Snap for every project just takes too much time. But there are some things you need sandboxes for, which brings us to this point.

GHC speedups are of course always welcome, but I don't think they will ever be good enough to solve this problem.

Collaborator

mightybyte commented Feb 13, 2013

Bummer. The case I'm talking about is really just when I'm creating a new sandbox. I've got this tree of a bunch of the dependencies I need, and they're already built. It just seems a shame to not be able to use them. Frankly, this is why I've always shied away from build sandboxes in the past--rebuilding a big stack like Snap for every project just takes too much time. But there are some things you need sandboxes for, which brings us to this point.

GHC speedups are of course always welcome, but I don't think they will ever be good enough to solve this problem.

@ghost ghost assigned 23Skidoo Feb 13, 2013

@23Skidoo

This comment has been minimized.

Show comment
Hide comment
@23Skidoo

23Skidoo Feb 13, 2013

Member

One approach is to use two package DBs: a shared cache for the common packages, and a project-specific package DB for things like unreleased dependencies and packages not in the common cache.

I think that something like this could work (not tested):

cd /path/to/shared-package-db
cabal sandbox-init
cabal sandbox-install shared-package-0 .. shared-package-N
cd /path/to/my-package
cabal sandbox-init

And then set the package-db field in cabal.sandbox.config to /path/to/shared-package-db/.cabal-sandbox/packages-$GHC_VER.conf:/path/to/my-package/.cabal-sandbox/packages-$GHC_VER.conf.

Member

23Skidoo commented Feb 13, 2013

One approach is to use two package DBs: a shared cache for the common packages, and a project-specific package DB for things like unreleased dependencies and packages not in the common cache.

I think that something like this could work (not tested):

cd /path/to/shared-package-db
cabal sandbox-init
cabal sandbox-install shared-package-0 .. shared-package-N
cd /path/to/my-package
cabal sandbox-init

And then set the package-db field in cabal.sandbox.config to /path/to/shared-package-db/.cabal-sandbox/packages-$GHC_VER.conf:/path/to/my-package/.cabal-sandbox/packages-$GHC_VER.conf.

@mightybyte

This comment has been minimized.

Show comment
Hide comment
@mightybyte

mightybyte Feb 13, 2013

Collaborator

Oh, that sounds almost exactly like what I'm talking about, except I was thinking more along the lines of:

cabal sandbox-init --foreign-package-db=/path/to/shared-package-db

...where the behavior of that flag would be that it causes necessary packages from the shared db to be copied in. This sounds structurally very similar to what I'm already doing except that I'm using cabal's default ~/.cabal repository as my shared DB. I'll play around with the package-db field in my cabal.sandbox.config and see if it does the job.

Collaborator

mightybyte commented Feb 13, 2013

Oh, that sounds almost exactly like what I'm talking about, except I was thinking more along the lines of:

cabal sandbox-init --foreign-package-db=/path/to/shared-package-db

...where the behavior of that flag would be that it causes necessary packages from the shared db to be copied in. This sounds structurally very similar to what I'm already doing except that I'm using cabal's default ~/.cabal repository as my shared DB. I'll play around with the package-db field in my cabal.sandbox.config and see if it does the job.

@tibbe

This comment has been minimized.

Show comment
Hide comment
@tibbe

tibbe Feb 13, 2013

Member

...where the behavior of that flag would be that it causes necessary packages from the shared db to be copied in.

This won't work. You have no guarantee that this package are usable with your current environment (e.g. they might have been built against the wrong dependencies). They may have to be rebuilt and thus break other packages in the shared package DB, which brings you back exactly where you were without sandboxing. If you want a shared package DB just don't use sandboxing!

Member

tibbe commented Feb 13, 2013

...where the behavior of that flag would be that it causes necessary packages from the shared db to be copied in.

This won't work. You have no guarantee that this package are usable with your current environment (e.g. they might have been built against the wrong dependencies). They may have to be rebuilt and thus break other packages in the shared package DB, which brings you back exactly where you were without sandboxing. If you want a shared package DB just don't use sandboxing!

@mightybyte

This comment has been minimized.

Show comment
Hide comment
@mightybyte

mightybyte Feb 13, 2013

Collaborator

The current environment is empty (hence the sandbox-init), so I don't see why you can't tell it to start with another environment that you know isn't broken.

Collaborator

mightybyte commented Feb 13, 2013

The current environment is empty (hence the sandbox-init), so I don't see why you can't tell it to start with another environment that you know isn't broken.

@23Skidoo

This comment has been minimized.

Show comment
Hide comment
@23Skidoo

23Skidoo Feb 19, 2013

Member

@mightybyte

The current environment is empty (hence the sandbox-init), so I don't see why you can't tell it to start with another environment that you know isn't broken.

In that case, can't you just do rm -rf .cabal-sandbox && cp -r old/.cabal-sandbox .cabal-sandbox? It'd be of course easy to implement sandbox init --copy-sandbox, but I think it's better to keep the UI as simple as possible for now.

Member

23Skidoo commented Feb 19, 2013

@mightybyte

The current environment is empty (hence the sandbox-init), so I don't see why you can't tell it to start with another environment that you know isn't broken.

In that case, can't you just do rm -rf .cabal-sandbox && cp -r old/.cabal-sandbox .cabal-sandbox? It'd be of course easy to implement sandbox init --copy-sandbox, but I think it's better to keep the UI as simple as possible for now.

@mightybyte

This comment has been minimized.

Show comment
Hide comment
@mightybyte

mightybyte Feb 19, 2013

Collaborator

Yeah, I'm fine with that. But in my case the sandbox I'm copying from is the default ~/.cabal repository, and I didn't know if it was the same format and could be copied wholesale with cp.

Collaborator

mightybyte commented Feb 19, 2013

Yeah, I'm fine with that. But in my case the sandbox I'm copying from is the default ~/.cabal repository, and I didn't know if it was the same format and could be copied wholesale with cp.

@23Skidoo

This comment has been minimized.

Show comment
Hide comment
@23Skidoo

23Skidoo Feb 19, 2013

Member

With ~/.cabal it's a bit more complicated since package.conf.d is under ~/.ghc. Now that I thought of it, you will also have to fix paths in *.conf files and do ghc-pkg recache in addition to copying.

Member

23Skidoo commented Feb 19, 2013

With ~/.cabal it's a bit more complicated since package.conf.d is under ~/.ghc. Now that I thought of it, you will also have to fix paths in *.conf files and do ghc-pkg recache in addition to copying.

@mightybyte

This comment has been minimized.

Show comment
Hide comment
@mightybyte

mightybyte Feb 19, 2013

Collaborator

Ahh, yes. So that makes at least two good reasons to have a feature like this. I don't know all the issues involved, but it seems like it might be nice to just make sandboxes the default behavior and have cabal traverse up the directory hierarchy until it finds a sandbox. I guess that could cause problems for people depending on the ~/.ghc repo. The line between GHC and Cabal gets rather fuzzy here.

Collaborator

mightybyte commented Feb 19, 2013

Ahh, yes. So that makes at least two good reasons to have a feature like this. I don't know all the issues involved, but it seems like it might be nice to just make sandboxes the default behavior and have cabal traverse up the directory hierarchy until it finds a sandbox. I guess that could cause problems for people depending on the ~/.ghc repo. The line between GHC and Cabal gets rather fuzzy here.

@christiaanb

This comment has been minimized.

Show comment
Hide comment
@christiaanb

christiaanb Mar 14, 2015

Collaborator

It was pointed out to me that I should perhaps leave a comment in this thread. Cabal 1.22 is shipped with preliminary support for so-called relocatable (or prefix-independent) packages. What his currently enables you to do is to have a completely relocatable .cabal-sandbox directory. This means you can move/copy the entire .cabal-sandbox directory anywhere you want on your machine.
To get a relocatable .cabal-sandbox dir just do:

  • cabal sandbox init
  • cabal install --dependencies-only --enable-relocatable

I know this doesn't get you completely what you describe in the issue, but I'm sure this new aspect of sandboxes will help in ultimately achieving it.

Collaborator

christiaanb commented Mar 14, 2015

It was pointed out to me that I should perhaps leave a comment in this thread. Cabal 1.22 is shipped with preliminary support for so-called relocatable (or prefix-independent) packages. What his currently enables you to do is to have a completely relocatable .cabal-sandbox directory. This means you can move/copy the entire .cabal-sandbox directory anywhere you want on your machine.
To get a relocatable .cabal-sandbox dir just do:

  • cabal sandbox init
  • cabal install --dependencies-only --enable-relocatable

I know this doesn't get you completely what you describe in the issue, but I'm sure this new aspect of sandboxes will help in ultimately achieving it.

@mietek

This comment has been minimized.

Show comment
Hide comment
@mietek

mietek Mar 14, 2015

Contributor

@mightybyte: Halcyon supports building sandboxes based on previously-built sandboxes.

See the Halcyon tutorial for an example.

This is made possible by keeping the sandbox in a pre-defined sandbox directory, and swapping the contents of this directory between projects. The trade-off is, only one build can be performed concurrently.

@christiaanb: Support for relocatable packages is a well-appreciated step forward. Thank you.

Contributor

mietek commented Mar 14, 2015

@mightybyte: Halcyon supports building sandboxes based on previously-built sandboxes.

See the Halcyon tutorial for an example.

This is made possible by keeping the sandbox in a pre-defined sandbox directory, and swapping the contents of this directory between projects. The trade-off is, only one build can be performed concurrently.

@christiaanb: Support for relocatable packages is a well-appreciated step forward. Thank you.

@ttuegel ttuegel modified the milestones: cabal-install-1.24, cabal-install-1.22 Apr 23, 2015

@23Skidoo 23Skidoo modified the milestones: cabal-install 1.24, cabal-install 1.26 Feb 21, 2016

@ezyang ezyang removed this from the cabal-install 2.0 milestone Sep 6, 2016

@ezyang ezyang removed this from the cabal-install 2.0 milestone Sep 6, 2016

@23Skidoo 23Skidoo removed their assignment Sep 13, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment