Merge nix-shell and build environments of haskell derivations

[bennofs]

This is an experiment to merge the .env and the normal build environment of packages by using ghcWithPackages for building too. The duplication of store paths is avoided by building the wrapper in a temporary directory directly while building the derivation itself. The advantages of this approach are:

• You can use nix-shell on normal derivations and don't need to deal with the .env attribute
• There now is only one approach in Nixpkgs' haskell packaging to make GHC find libraries: building a symlinked package db. This ensures that there is no package that builds but doesn't work with the wrapper.
• We don't need GHC_PACKAGE_PATH, which doesn't work well with cabal-install

I've successfully build all packages from https://github.com/peti/ci/blob/master/haskell-nixpkgs.nix with big = false with the new builder. Of course this means that any changes to the wrapper also require recompilation of all haskell packages, but the wrapper doesn't change that often anyway. The wrapper for the old haskellPackages set was changed last in November 2013, for example, which is over a year ago.

/cc @peti and anyone else interrested in haskell nixpkgs packaging.

[jagajaga]

Sorry if it is a stupid a question, but what changes, what opportunities are we getting? :) I really can't get it from the description.

[bennofs]

@jagajaga You don't need a separate .env attribute for haskell derivations anymore. This means that you can use nix-build and nix-shell on the same derivation, and also that the build environment more closely mirros the nix-shell environment

[jagajaga]

@bennofs oh! got it. Looks nice. 👍

[peti]

I implemented the same feature in bbb109c a while ago: instead of creating a custom package.conf.d database locally for the build, the generic builder would re-use ghcWithPackages to create a wrapped ghc variant that knew all required packages. As you can see from the commit message, I felt enthusiastic about that change. 😄

The design goals I hoped to achieve were:

1. Simpler code in the generic builder.
2. Allow builds to bake the full path of ghc into their code to obtain a build environment in which they can re-compile themselves, like xmonad and friends do.
3. Unify the normal nix-build process with the interactive nix-shell environment.

As it happens, the new code didn't meet those goals entirely.

Regarding (1), the code did become simpler, but only marginally so, because some logic is still required to compute the proper set of buildInputs, the proper --extra-include-dirs and --extra-lib-dirs flags for Cabal, etc. Re-using the wrapper saved a handful of lines here and there, but IMHO it didn't make the structure of the builder much simpler.

Goal (2) came through, of course, but at a price: with the new builder, every Haskell build required an additional build to set up its compiler environment. In other words, an attempt to build 1 derivation actually built 2, littering /nix/store with thousands of ghc environments within a couple of hours. Arguably, most of those derivations where build-time-only dependencies that Nix can garbage-collect, but still this kind of wastefulness felt a little disconcerting.

Last but not least, goal (3) did work out, but it became apparent that there is trouble on the horizon. In my attempts writing the generic builder, I tried hard to leave buildInputs empty. The reason is that ghc is not supposed to find system libraries automatically; instead we must call cabal configure with the proper search paths to make sure Cabal bakes those paths into the generated package.conf file. Otherwise, we run the risk that those builds are unusable outside of the Nix build environment, i.e. when people use a ghcWithPackages environment in their normal shell. It turned out, however, that I cannot achieve an empty buildInputs attribute, because our build specifications aren't accurate enough to compute all required settings properly. I'm currently addressing this issue is in the wip branch:

$ cabal2nix cabal://gtk3
{ mkDerivation, array, base, bytestring, cairo, containers, gio
, glib, gtk2hs-buildtools, gtk3, mtl, pango, stdenv, text, time
, transformers
}:
mkDerivation {
  pname = "gtk3";
  version = "0.13.4";
  sha256 = "041yg1h7g7mm3iy6wz8mp74ximw63vc4x918gpha5lpj6l2fxaqh";
  isLibrary = true;
  isExecutable = true;
  libraryDepends = { haskell = [array base bytestring cairo containers gio glib mtl pango text];
                     system = [ gtk2hs-buildtools ]; pkgconfig = [ glib gtk3 ];
                   };
  executableDepends = { haskell = [array base cairo gtk3 text time transformers]; };
  homepage = "http://projects.haskell.org/gtk2hs/";
  description = "Binding to the Gtk+ graphical user interface library";
  license = stdenv.lib.licenses.lgpl21;
}

The new build description distinguishes nicely between Haskell library dependencies (that ghc ought to know about by means of package.conf.d), system libraries (that Cabal ought to know about by means of --extra-include-dirs and --extra-lib-dirs), package config dependencies (that Cabal ought to know about by means of calling pkg-config at run-time), etc. With that information, we'll be able to run build in an (almost) empty environment, i.e. we pass all necessary information to cabal configure instead of relying on $NIX_CFLAGS magic and its like. IMHO, that change will make our builds much more robust.

Now, the problem with regard to unifying nix-build and nix-shell is that nix-shell can, in general, not call cabal configure for you. If you call cabal configure yourself, however, how do you pass all those required flags? Right now, the env environment "just works" because we use $NIX_CFLAGS style magic to let ghc find system dependencies automatically without being told about their location. That's fine (and actually desirable) for an interactive development environment. In non-interactive nix-builds, however, we don't want that!

At some point I began to wonder whether the fact that those two build styles had separate attributes (which allows the user to choose the one she prefers) is a good thing, actually.

Furthermore, I have some ideas for the ghcWithPackages wrapper that make perfect sense in user environments -- but they don't make sense for non-interactive builds. As an example, I'd like to add code that wraps all binaries in $out/bin to ensure they set the required $NIX_GHC variables. This can be useful for programs like ghc-mod. The feature is nice for a user environment, but it's pointless inside of nix-build because we control the shell environment in there, so these wrappers don't achieve anything. My point is that re-using the wrapper inside of generic builds saves some complexity, but it also adds some complexity, because sooner or later we'll end up adding conditional logic to the wrapper to distinguish whether it's used in one case or the other.

Now, your pull request chooses a slightly different path than my implementation, because it doesn't request the ghcWithPackages environment from Nix. Instead, it builds that environment in a temporary location. This technique avoids the plethora of derivations that my code generated, which is great! At the same time, this approach breaks design goal (2), because now builds run with a ghc binary that lives in a transient location and won't exist anymore after the build is complete.

As far as the other pros and cons are concerned, both your and my implementation seem to be about the same.

[bennofs]

Thank you very much for this extremly detailed response! I see now why you choose to go the path of an .env attribute.

So this PR can be closed then.