Construct a sandbox from another package repository #1196

Open
mightybyte opened this Issue Feb 13, 2013 · 16 comments

Projects

None yet

7 participants

@mightybyte
Collaborator

I routinely work on multiple projects that depend on each other but that I don't want to share a sandbox. It would be a huge time saver if there was a way to do "cabal sandbox-install" but have it get already installed packages from the old cabal --user or --global repositories (or maybe even other sandboxes) without having to rebuild them. In my use case, I would want the package to be copied to the new repository so that this sandbox still works independent of the other repository.

@tibbe
Member
tibbe commented Feb 13, 2013

The sandbox-add-source <path> command (which will most likely be cabal sandbox add-source <path> in the release) is meant to let you depend on unreleased packages that you have checked-out in e.g. $HOME/src. add-source creates a link such that the dependency gets rebuilt if it changes. We also plan to add a --snapshot flag which would install a copy of the dependency (just like cabal-dev's add-source does).

We're generally trying to move away from installing things. Projects should be self-contained and build all their dependencies in a way that they don't clobber other projects (which is what shared, mutable package stores result in).

@mightybyte
Collaborator

I'm using add-source. But here I'm talking about packages coming from hackage, not add-source.

@tibbe
Member
tibbe commented Feb 13, 2013

I misunderstood. You want the build times to be lower. We will have to wait for a Nix-style package store for that to happen. In the meantime, add jobs: $ncpus to your $HOME/.cabal/config file.

@mightybyte
Collaborator

Yeah, I've done that. Build times are still long. I guess I may not appreciate all the details, but it doesn't seem too difficult to add a flag that would include that packages from another repository in the list of things it thinks it already has.

@tibbe
Member
tibbe commented Feb 13, 2013

Not much we can do at this point I'm afraid. We cannot safely just copy installed packages around (that's why we need a Nix-style store that correctly tracks inputs so outputs can be used). Between that and making GHC faster there's not much else.

@mightybyte
Collaborator

Bummer. The case I'm talking about is really just when I'm creating a new sandbox. I've got this tree of a bunch of the dependencies I need, and they're already built. It just seems a shame to not be able to use them. Frankly, this is why I've always shied away from build sandboxes in the past--rebuilding a big stack like Snap for every project just takes too much time. But there are some things you need sandboxes for, which brings us to this point.

GHC speedups are of course always welcome, but I don't think they will ever be good enough to solve this problem.

@23Skidoo 23Skidoo was assigned Feb 13, 2013
@23Skidoo
Member

One approach is to use two package DBs: a shared cache for the common packages, and a project-specific package DB for things like unreleased dependencies and packages not in the common cache.

I think that something like this could work (not tested):

cd /path/to/shared-package-db
cabal sandbox-init
cabal sandbox-install shared-package-0 .. shared-package-N
cd /path/to/my-package
cabal sandbox-init

And then set the package-db field in cabal.sandbox.config to /path/to/shared-package-db/.cabal-sandbox/packages-$GHC_VER.conf:/path/to/my-package/.cabal-sandbox/packages-$GHC_VER.conf.

@mightybyte
Collaborator

Oh, that sounds almost exactly like what I'm talking about, except I was thinking more along the lines of:

cabal sandbox-init --foreign-package-db=/path/to/shared-package-db

...where the behavior of that flag would be that it causes necessary packages from the shared db to be copied in. This sounds structurally very similar to what I'm already doing except that I'm using cabal's default ~/.cabal repository as my shared DB. I'll play around with the package-db field in my cabal.sandbox.config and see if it does the job.

@tibbe
Member
tibbe commented Feb 13, 2013

...where the behavior of that flag would be that it causes necessary packages from the shared db to be copied in.

This won't work. You have no guarantee that this package are usable with your current environment (e.g. they might have been built against the wrong dependencies). They may have to be rebuilt and thus break other packages in the shared package DB, which brings you back exactly where you were without sandboxing. If you want a shared package DB just don't use sandboxing!

@mightybyte
Collaborator

The current environment is empty (hence the sandbox-init), so I don't see why you can't tell it to start with another environment that you know isn't broken.

@23Skidoo
Member

@mightybyte

The current environment is empty (hence the sandbox-init), so I don't see why you can't tell it to start with another environment that you know isn't broken.

In that case, can't you just do rm -rf .cabal-sandbox && cp -r old/.cabal-sandbox .cabal-sandbox? It'd be of course easy to implement sandbox init --copy-sandbox, but I think it's better to keep the UI as simple as possible for now.

@mightybyte
Collaborator

Yeah, I'm fine with that. But in my case the sandbox I'm copying from is the default ~/.cabal repository, and I didn't know if it was the same format and could be copied wholesale with cp.

@23Skidoo
Member

With ~/.cabal it's a bit more complicated since package.conf.d is under ~/.ghc. Now that I thought of it, you will also have to fix paths in *.conf files and do ghc-pkg recache in addition to copying.

@mightybyte
Collaborator

Ahh, yes. So that makes at least two good reasons to have a feature like this. I don't know all the issues involved, but it seems like it might be nice to just make sandboxes the default behavior and have cabal traverse up the directory hierarchy until it finds a sandbox. I guess that could cause problems for people depending on the ~/.ghc repo. The line between GHC and Cabal gets rather fuzzy here.

@23Skidoo 23Skidoo added the enhancement label Mar 8, 2014
@christiaanb
Collaborator

It was pointed out to me that I should perhaps leave a comment in this thread. Cabal 1.22 is shipped with preliminary support for so-called relocatable (or prefix-independent) packages. What his currently enables you to do is to have a completely relocatable .cabal-sandbox directory. This means you can move/copy the entire .cabal-sandbox directory anywhere you want on your machine.
To get a relocatable .cabal-sandbox dir just do:

  • cabal sandbox init
  • cabal install --dependencies-only --enable-relocatable

I know this doesn't get you completely what you describe in the issue, but I'm sure this new aspect of sandboxes will help in ultimately achieving it.

@mietek
Contributor
mietek commented Mar 14, 2015

@mightybyte: Halcyon supports building sandboxes based on previously-built sandboxes.

See the Halcyon tutorial for an example.

This is made possible by keeping the sandbox in a pre-defined sandbox directory, and swapping the contents of this directory between projects. The trade-off is, only one build can be performed concurrently.

@christiaanb: Support for relocatable packages is a well-appreciated step forward. Thank you.

@ezyang ezyang modified the milestone: cabal-install 2.0 Sep 6, 2016
@23Skidoo 23Skidoo removed their assignment Sep 13, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment