Make Cabal sandboxes relocatable

[2piix]

This is a minor feature request. I've done some experimentation and it all seems to work, but I haven't read or written any Cabal source code.

I would like Cabal sandboxes to be relocatable. Effectively, this means that the sandbox would be unaware of its location in the file system. As far as I know, this should not be a problem, since the sandbox only points to files inside its container.

Here are some use-cases:

• pre-built sandboxes, ready for development. Sandboxes are super-cool, and I have been enjoying them since before the state of the Haskell union survey that one year when I suggested cabal-dev and everybody listened. (Sorry, I am proud of my very minor contribution to computing history. :-) But there's no denying that having to wait for sandboxes to build all the time is not fun. But if we can just build a sandbox and clone it when we need it, we can pay for it once. This would be especially useful for projects with lots of "movement":

  do
  darcs clone
  cabal sandbox init
  cabal install

  do
  cp sandbox
  fix bug
  darcs push

• making an OS X bundle seems to be a plist file away. Make the plist point at dist/bin/foo, and bob's your uncle.

I tried an experiment where I built a sandbox and set cabal.config to point to .cabal-sandbox instead of /home/foo/bar/baz/.cabal-sandbox. Afterwards, I made a copy of the sandbox, went into it, and ran cabal repl. It worked perfectly.

It seems to me (keeping in mind that I have not read sources and defer to your opinion) that all we'd have to do is change the default sandbox configuration emitter to not prefix directory names with the current working directory. Is that sensible? Is there something else I'm missing?

[jarlg]

@2piix Yes, cabal repl seems to work after making the paths relative, but most things else break (try a cabal install if you're not convinced). See #1490 and #1542.

[2piix]

I see, thank you. I still want it, but I see it isn't a simple change. :)

[christiaanb]

It was pointed out to me that I should perhaps leave a comment in this thread. Cabal 1.22 is shipped with preliminary support for so-called relocatable (or prefix-independent) packages. What his currently enables you to do is to have a completely relocatable .cabal-sandbox directory. This means you can move/copy the entire .cabal-sandbox directory anywhere you want on your machine.
To get a relocatable .cabal-sandbox dir just do:

• cabal sandbox init
• cabal install --dependencies-only --enable-relocatable

I know this doesn't get you completely what you describe in the issue, but I'm sure this new aspect of sandboxes will help in ultimately achieving it.

[BardurArantsson]

Is there anything actionable left in this issue or can it be closed?

[bmjoan]

@BardurArantsson Some packages with external FFI library dependencies fail to install as relocatable in sandboxes.

For example cabal install --enable-relocatable postgresql-simple unexpectedly fails with

dependency: "/usr/lib64/postgresql/9.3/lib"
is not relative to the installation prefix:
"/home/joanbm/Projects/Haskell/playground/.cabal-sandbox"
cabal: Error: some packages failed to install:
postgresql-simple-0.4.10.0 failed during the configure step. The exception
was:
ExitFailure 1

The path to PostgreSQL's libpq.so is get from pkg-config.

I see no reason why full paths to external libraries are not allowed.

[BardurArantsson]

@joanblake I think that would be a separate (more "focused") issue... would you mind creating an issue for that?

(Though, I think it might somewhat go against the idea of relocatability to use absolute paths... but then again external dependencies technically aren't part of the sandbox, so...)

[christiaanb]

@joanblake Indeed, when I created partial support for prefix-independent packages, it didn't cross my mind that there might be external, non Haskell, libraries that a package could depend on. I don't know if we can currently distinguish between the library paths of Haskell and non-Haskell libraries. I guess creating a separate issue indeed makes sense.

@BardurArantsson This issue cannot be closed, because the cabal-sandbox.config files are not prefix-independent. They still contain absolute paths.

[BardurArantsson]

@christiaanb Ah, I forgot about that. If that's the only thing missing, perhaps create a specific issue for that bit? (Unless of course you're actively working on this and we're just a PR away... in which case creating a different issue would be pointless.)

My thinking here is that issues should be as specific (i.e. non-vague) as possible to encourage others (who may not familiar with the overall plan) to be able to contribute.

[BenBals]

I recently had to move a Haskell project featuring a cabal sandbox and updating all paths in cabal.sandbox.config using a simple find and replace in a text editor fixed all errors for me. Well at least cabal run still works fine. Will have to see about installing new dependencies.

[bradrn]

I know this is a very old issue, but: does anyone know whether this is working on Windows? When I try I get an error:

C:\Users\bradn\Documents\Haskell>mkdir cabal-sandbox-test

C:\Users\bradn\Documents\Haskell>cd cabal-sandbox-test

C:\Users\bradn\Documents\Haskell\cabal-sandbox-test>stack exec --no-ghc-package-path -- cabal sandbox init
Warning: The sandbox command is a part of the legacy v1 style of cabal usage.

Please switch to using either the new project style and the new-sandbox
command or the legacy v1-sandbox alias as new-style projects will become the
default in the next version of cabal-install. Please file a bug if you cannot
replicate a working v1- use case with the new-style commands.

For more information, see: https://wiki.haskell.org/Cabal/NewBuild

Writing a default package environment file to
C:\Users\bradn\Documents\Haskell\cabal-sandbox-test\cabal.sandbox.config
Creating a new sandbox at
C:\Users\bradn\Documents\Haskell\cabal-sandbox-test\.cabal-sandbox

C:\Users\bradn\Documents\Haskell\cabal-sandbox-test>stack exec --no-ghc-package-path -- cabal install --enable-relocatable containers
Warning: The install command is a part of the legacy v1 style of cabal usage.

Please switch to using either the new project style and the new-install
command or the legacy v1-install alias as new-style projects will become the
default in the next version of cabal-install. Please file a bug if you cannot
replicate a working v1- use case with the new-style commands.

For more information, see: https://wiki.haskell.org/Cabal/NewBuild

Warning: Parsing the index cache failed (Unknown encoding for constructor).
Trying to regenerate the index cache...
Resolving dependencies...
Notice: installing into a sandbox located at
C:\Users\bradn\Documents\Haskell\cabal-sandbox-test\.cabal-sandbox
Downloading  containers-0.6.0.1
Downloaded   containers-0.6.0.1
Starting     containers-0.6.0.1
Failed to install containers-0.6.0.1
Build log ( C:\Users\bradn\Documents\Haskell\cabal-sandbox-test\.cabal-sandbox\logs\ghc-8.4.3\containers-0.6.0.1-6dUFjaTrYaoIACE6pzeQpO.log ):
Configuring containers-0.6.0.1...
cabal.EXE: Operating system: windows, does not support relocatable builds

cabal: Leaving directory 'C:\Users\bradn\AppData\Local\Temp\cabal-tmp-1000\containers-0.6.0.1'
cabal.EXE: Error: some packages failed to install:
containers-0.6.0.1-6dUFjaTrYaoIACE6pzeQpO failed during the configure step.
The exception was:
ExitFailure 1

Note the line:

cabal.EXE: Operating system: windows, does not support relocatable builds

[christiaanb]

@bradrn Apologies for the late response: no, --enable-relocatable doesn't work on Windows because I didn't know how to do relative search-paths for dynamic libraries on Windows; in Linux you add RPATHs to the dynamic linker section so that the run-time linker knows where to search for the libraries objects (.so on Linux, .dll on Windows) the library or executable depends on.

[Mistuke]

I'm confused though, where does RPATH come into play here? Windows is fully statically linked, so there are no DLLs for it to depend on. How was this solved for non-dynamicWay builds of GHC on Linux? There RPATH wouldn't work either and would have the same problem.

In any case the RPATH equivalent on Windows can be achieved by using delay [1] loading along with the exposed helper functions[2].

[1] https://docs.microsoft.com/en-us/cpp/build/reference/linker-support-for-delay-loaded-dlls?view=vs-2019
[2] https://docs.microsoft.com/en-us/cpp/build/reference/understanding-the-helper-function?view=vs-2019

[bradrn]

@christiaanb That's a pity - relocatable sandboxes would have been quite useful for what I'm doing.

I am wondering something though: what exactly do dynamic libraries have to do with relocatable sandboxes? I don't see how the two are related.

[christiaanb]

Right... this is/was a mistake on my end:

1. On most non-windows platforms, dynamic linking is the default
2. In order for dynamic linking to work, the run-time search paths for libraries needs to be prefix-independent
3. By default, GHC tells the linker to add absolute paths for dynamic dependencies
4. However, you can override this behaviour by supplying rpaths manually (for which GHC has a flag), which is what Cabal does when it calls GHC to link the final library/executable. It can provide prefix-independent rpaths because unlike GHC, Cabal knows where the library/executable will be installed.

So two things:

1. Given that static linking is the default on windows, relocatable builds can probably be enabled. You just have to change the following check:

   cabal/Cabal/Distribution/Simple/Configure.hs

   Lines 1928 to 1938 in 33c271e

   	-- Check if the OS support relocatable builds.
   	--
   	-- If you add new OS' to this list, and your OS supports dynamic libraries
   	-- and RPATH, make sure you add your OS to RPATH-support list of:
   	-- Distribution.Simple.GHC.getRPaths
   	checkOS
   	= unless (os `elem` [ OSX, Linux ])
   	$ die' verbosity $ "Operating system: " ++ prettyShow os ++
   	", does not support relocatable builds"
   	where
   	(Platform _ os) = hostPlatform lbi

   
   So that windows is accepted as a platform on which relocatable builds work, as long as you are building a statically linked library/executable. You'd probably also want to update the comments, saying the RPATH business doesn't apply to Windows because it does static linking by default.
2. @Mistuke Do you know whether GHC supports building DLLs/EXEs with that delay loading you mentioned?

[Mistuke]

@christiaanb GHC calls GCC by default with -Wl,--out-implib which builds the eager binding version. dlltool does support the lazy one but it needs to be a multi step thing. You build the dll, build the export definition and then build the lazy import library using dlltool.

[jneira]

No more sandboxes, so i think this one could be closed (and the other ones labeled with sandbox maybe too)
//cc @phadej

[phadej]

@jneira, indeed! thanks for mentioning that. Let me see what else I can close.

[phadej]

The issue bar now says 1.3k, not 1.4k. Thanks, @jneira, that made my day!