Skip to content

Commit

Permalink
Back-port Haskell-related improvements from stdenv-updates.
Browse files Browse the repository at this point in the history 
 * There now is full support for building Haskell packages as shared libraries
   for GHC versions 7.4.2 or later. The Cabal builder recognizes the following
   attributes:

    - enableSharedLibraries configures Cabal to build of shared libraries in
      addition to static ones. This option requires that all dependencies of
      the package have been compiled for use in shared libraries, too.

    - enableSharedExecutables configures Cabal to prefer shared libraries when
      linking executables.

   The default values for these attributes are arguments to the haskellPackages
   expression.

 * Haskell builds now run in a LANG="en_US.UTF-8" environment to avoid plenty
   of build and test suite errors. Without this setting, GHC seems unable to
   deal with the UTF-8 character encoding that's generally considered standard
   in the Haskell world.

 * The Cabal builder supports a new attribute 'testTarget' to specify the exact
   set of tests to be run during the check phase.

 * The ghc-wrapper attribute ghcVersion has been removed. Instead, we use the
   ghc.version attribute, which exists in unwrapped GHC derivations, too.
  • Loading branch information
@peti
peti committed Oct 27, 2013
1 parent 5580abd commit d64917a
Show file tree
Hide file tree
Showing 21 changed files with 184 additions and 211 deletions.
93 changes: 67 additions & 26 deletions pkgs/build-support/cabal/default.nix
View file
@@ -1,12 +1,29 @@
# generic builder for Cabal packages

{ stdenv, fetchurl, lib, pkgconfig, ghc, Cabal, jailbreakCabal
{ stdenv, fetchurl, lib, pkgconfig, ghc, Cabal, jailbreakCabal, glibcLocales
, enableLibraryProfiling ? false
, enableCheckPhase ? true
, enableSharedLibraries ? false
, enableSharedExecutables ? false
, enableCheckPhase ? stdenv.lib.versionOlder "7.4" ghc.version
}:

# The Cabal library shipped with GHC versions older than 7.x doesn't accept the --enable-tests configure flag.
assert enableCheckPhase -> stdenv.lib.versionOlder "7" ghc.ghcVersion;
let
enableFeature = stdenv.lib.enableFeature;
versionOlder = stdenv.lib.versionOlder;
optional = stdenv.lib.optional;
optionals = stdenv.lib.optionals;
optionalString = stdenv.lib.optionalString;
filter = stdenv.lib.filter;
in

# Cabal shipped with GHC 6.12.4 or earlier doesn't know the "--enable-tests configure" flag.
assert enableCheckPhase -> versionOlder "7" ghc.version;

# GHC prior to 7.4.x doesn't know the "--enable-executable-dynamic" flag.
assert enableSharedExecutables -> versionOlder "7.4" ghc.version;

# Our GHC 6.10.x builds do not provide sharable versions of their core libraries.
assert enableSharedLibraries -> versionOlder "6.12" ghc.version;

{
mkDerivation =
Expand All @@ -23,8 +40,8 @@ assert enableCheckPhase -> stdenv.lib.versionOlder "7" ghc.ghcVersion;
# in the interest of keeping hashes stable.
postprocess =
x : (removeAttrs x internalAttrs) // {
buildInputs = stdenv.lib.filter (y : ! (y == null)) x.buildInputs;
propagatedBuildInputs = stdenv.lib.filter (y : ! (y == null)) x.propagatedBuildInputs;
buildInputs = filter (y : ! (y == null)) x.buildInputs;
propagatedBuildInputs = filter (y : ! (y == null)) x.propagatedBuildInputs;
doCheck = enableCheckPhase && x.doCheck;
};

Expand All @@ -42,8 +59,12 @@ assert enableCheckPhase -> stdenv.lib.versionOlder "7" ghc.ghcVersion;
# if that is not desired (for applications), name can be set to
# fname.
name = if self.isLibrary then
if enableLibraryProfiling then
if enableLibraryProfiling && self.enableSharedLibraries then
"haskell-${self.pname}-ghc${ghc.ghc.version}-${self.version}-profiling-shared"
else if enableLibraryProfiling && !self.enableSharedLibraries then
"haskell-${self.pname}-ghc${ghc.ghc.version}-${self.version}-profiling"
else if !enableLibraryProfiling && self.enableSharedLibraries then
"haskell-${self.pname}-ghc${ghc.ghc.version}-${self.version}-shared"
else
"haskell-${self.pname}-ghc${ghc.ghc.version}-${self.version}"
else
Expand All @@ -63,7 +84,7 @@ assert enableCheckPhase -> stdenv.lib.versionOlder "7" ghc.ghcVersion;
# but often propagatedBuildInputs is preferable anyway
buildInputs = [ghc Cabal] ++ self.extraBuildInputs;
extraBuildInputs = self.buildTools ++
(stdenv.lib.optionals self.doCheck self.testDepends) ++
(optionals self.doCheck self.testDepends) ++
(if self.pkgconfigDepends == [] then [] else [pkgconfig]) ++
(if self.isLibrary then [] else self.buildDepends ++ self.extraLibraries ++ self.pkgconfigDepends);

Expand All @@ -80,6 +101,9 @@ assert enableCheckPhase -> stdenv.lib.versionOlder "7" ghc.ghcVersion;
# build-depends Cabal fields stated in test-suite stanzas
testDepends = [];

# target(s) passed to the cabal test phase as an argument
testTarget = "";

# build-tools Cabal field
buildTools = [];

Expand All @@ -96,42 +120,61 @@ assert enableCheckPhase -> stdenv.lib.versionOlder "7" ghc.ghcVersion;
jailbreak = false;

# pass the '--enable-split-objs' flag to cabal in the configure stage
enableSplitObjs = !( stdenv.isDarwin # http://hackage.haskell.org/trac/ghc/ticket/4013
|| stdenv.lib.versionOlder "7.6.99" ghc.ghcVersion # -fsplit-ojbs is broken in 7.7 snapshot
enableSplitObjs = !( stdenv.isDarwin # http://hackage.haskell.org/trac/ghc/ticket/4013
|| versionOlder "7.6.99" ghc.version # -fsplit-ojbs is broken in 7.7 snapshot
);

# pass the '--enable-tests' flag to cabal in the configure stage
# and run any regression test suites the package might have
doCheck = enableCheckPhase;

# pass the '--enable-shared' flag to cabal in the configure
# stage to enable building shared libraries
inherit enableSharedLibraries;

# pass the '--enable-executable-dynamic' flag to cabal in
# the configure stage to enable linking shared libraries
inherit enableSharedExecutables;

extraConfigureFlags = [
(stdenv.lib.enableFeature enableLibraryProfiling "library-profiling")
(stdenv.lib.enableFeature self.enableSplitObjs "split-objs")
] ++ stdenv.lib.optional (stdenv.lib.versionOlder "7" ghc.ghcVersion) (stdenv.lib.enableFeature self.doCheck "tests");
(enableFeature self.enableSplitObjs "split-objs")
(enableFeature enableLibraryProfiling "library-profiling")
(enableFeature self.enableSharedLibraries "shared")
(optional (versionOlder "7.4" ghc.version) (enableFeature self.enableSharedExecutables "executable-dynamic"))
(optional (versionOlder "7" ghc.version) (enableFeature self.doCheck "tests"))
];

# GHC needs the locale configured during the Haddock phase.
LANG = "en_US.UTF-8";
LOCALE_ARCHIVE = optionalString stdenv.isLinux "${glibcLocales}/lib/locale/locale-archive";

# compiles Setup and configures
configurePhase = ''
eval "$preConfigure"
${lib.optionalString self.jailbreak "${jailbreakCabal}/bin/jailbreak-cabal ${self.pname}.cabal"}
${optionalString self.jailbreak "${jailbreakCabal}/bin/jailbreak-cabal ${self.pname}.cabal"}
for i in Setup.hs Setup.lhs; do
test -f $i && ghc --make $i
done
for p in $extraBuildInputs $propagatedNativeBuildInputs; do
if [ -d "$p/lib/ghc-${ghc.ghc.version}/package.conf.d" ]; then
# Haskell packages don't need any extra configuration.
continue;
fi
if [ -d "$p/include" ]; then
extraConfigureFlags+=" --extra-include-dir=$p/include"
extraConfigureFlags+=" --extra-include-dirs=$p/include"
fi
for d in lib{,64}; do
if [ -d "$p/$d" ]; then
extraConfigureFlags+=" --extra-lib-dir=$p/$d"
extraConfigureFlags+=" --extra-lib-dirs=$p/$d"
fi
done
done
echo "configure flags: $extraConfigureFlags $configureFlags"
./Setup configure --verbose --prefix="$out" $extraConfigureFlags $configureFlags
./Setup configure --verbose --prefix="$out" --libdir='$prefix/lib/$compiler' --libsubdir='$pkgid' $extraConfigureFlags $configureFlags
eval "$postConfigure"
'';
Expand All @@ -142,16 +185,16 @@ assert enableCheckPhase -> stdenv.lib.versionOlder "7" ghc.ghcVersion;
./Setup build
export GHC_PACKAGE_PATH=$(ghc-packages)
[ -n "$noHaddock" ] || ./Setup haddock
export GHC_PACKAGE_PATH=$(${ghc.GHCPackages})
test -n "$noHaddock" || ./Setup haddock
eval "$postBuild"
'';

checkPhase = stdenv.lib.optional self.doCheck ''
checkPhase = optional self.doCheck ''
eval "$preCheck"
./Setup test
./Setup test ${self.testTarget}
eval "$postCheck"
'';
Expand All @@ -166,7 +209,7 @@ assert enableCheckPhase -> stdenv.lib.versionOlder "7" ghc.ghcVersion;
ensureDir $out/bin # necessary to get it added to PATH
local confDir=$out/lib/ghc-pkgs/ghc-${ghc.ghc.version}
local confDir=$out/lib/ghc-${ghc.ghc.version}/package.conf.d
local installedPkgConf=$confDir/${self.fname}.installedconf
local pkgConf=$confDir/${self.fname}.conf
ensureDir $confDir
Expand All @@ -176,13 +219,11 @@ assert enableCheckPhase -> stdenv.lib.versionOlder "7" ghc.ghcVersion;
GHC_PACKAGE_PATH=$installedPkgConf ghc-pkg --global register $pkgConf --force
fi
eval "$postInstall"
'';

postFixup = ''
if test -f $out/nix-support/propagated-native-build-inputs; then
ln -s $out/nix-support/propagated-native-build-inputs $out/nix-support/propagated-user-env-packages
fi
eval "$postInstall"
'';

# We inherit stdenv and ghc so that they can be used
Expand Down
4 changes: 2 additions & 2 deletions pkgs/development/compilers/ghc/6.10.1-binary.nix
View file
Expand Up @@ -36,10 +36,10 @@ stdenv.mkDerivation rec {
''
mkdir "$TMP/bin"
for i in strip; do
echo '#!/bin/sh' >> "$TMP/bin/$i"
echo '#! ${stdenv.shell}' > "$TMP/bin/$i"
chmod +x "$TMP/bin/$i"
PATH="$TMP/bin:$PATH"
done
PATH="$TMP/bin:$PATH"
'' +
# On Linux, use patchelf to modify the executables so that they can
# find editline/gmp.
Expand Down
4 changes: 2 additions & 2 deletions pkgs/development/compilers/ghc/6.10.2-binary.nix
View file
Expand Up @@ -30,10 +30,10 @@ stdenv.mkDerivation rec {
''
mkdir "$TMP/bin"
for i in strip; do
echo '#!/bin/sh' >> "$TMP/bin/$i"
echo '#! ${stdenv.shell}' > "$TMP/bin/$i"
chmod +x "$TMP/bin/$i"
PATH="$TMP/bin:$PATH"
done
PATH="$TMP/bin:$PATH"
'' +
# On Linux, use patchelf to modify the executables so that they can
# find editline/gmp.
Expand Down
4 changes: 2 additions & 2 deletions pkgs/development/compilers/ghc/6.12.1-binary.nix
View file
Expand Up @@ -28,10 +28,10 @@ stdenv.mkDerivation rec {
''
mkdir "$TMP/bin"
for i in strip; do
echo '#!/bin/sh' >> "$TMP/bin/$i"
echo '#! ${stdenv.shell}' > "$TMP/bin/$i"
chmod +x "$TMP/bin/$i"
PATH="$TMP/bin:$PATH"
done
PATH="$TMP/bin:$PATH"
'' +
# We have to patch the GMP paths for the integer-gmp package.
''
Expand Down
4 changes: 2 additions & 2 deletions pkgs/development/compilers/ghc/7.0.4-binary.nix
View file
Expand Up @@ -38,10 +38,10 @@ stdenv.mkDerivation rec {
''
mkdir "$TMP/bin"
for i in strip; do
echo '#!/bin/sh' >> "$TMP/bin/$i"
echo '#! ${stdenv.shell}' > "$TMP/bin/$i"
chmod +x "$TMP/bin/$i"
PATH="$TMP/bin:$PATH"
done
PATH="$TMP/bin:$PATH"
'' +
# We have to patch the GMP paths for the integer-gmp package.
''
Expand Down
4 changes: 2 additions & 2 deletions pkgs/development/compilers/ghc/7.4.2-binary.nix
View file
Expand Up @@ -38,10 +38,10 @@ stdenv.mkDerivation rec {
''
mkdir "$TMP/bin"
for i in strip; do
echo '#!/bin/sh' >> "$TMP/bin/$i"
echo '#! ${stdenv.shell}' > "$TMP/bin/$i"
chmod +x "$TMP/bin/$i"
PATH="$TMP/bin:$PATH"
done
PATH="$TMP/bin:$PATH"
'' +
# We have to patch the GMP paths for the integer-gmp package.
''
Expand Down
1 change: 1 addition & 0 deletions pkgs/development/compilers/ghc/7.4.2.nix
View file
Expand Up @@ -22,6 +22,7 @@ stdenv.mkDerivation rec {
preConfigure = ''
echo "${buildMK}" > mk/build.mk
sed -i -e 's|-isysroot /Developer/SDKs/MacOSX10.5.sdk||' configure
export NIX_LDFLAGS="$NIX_LDFLAGS -rpath $out/lib/ghc-${version}"
'';

configureFlags=[
Expand Down
5 changes: 2 additions & 3 deletions pkgs/development/compilers/ghc/7.6.3.nix
View file
Expand Up @@ -22,11 +22,10 @@ stdenv.mkDerivation rec {
preConfigure = ''
echo "${buildMK}" > mk/build.mk
sed -i -e 's|-isysroot /Developer/SDKs/MacOSX10.5.sdk||' configure
export NIX_LDFLAGS="$NIX_LDFLAGS -rpath $out/lib/ghc-${version}"
'';

configureFlags = [
"--with-gcc=${stdenv.gcc}/bin/gcc"
];
configureFlags = "--with-gcc=${stdenv.gcc}/bin/gcc";

# required, because otherwise all symbols from HSffi.o are stripped, and
# that in turn causes GHCi to abort
Expand Down
1 change: 1 addition & 0 deletions pkgs/development/compilers/ghc/head.nix
View file
Expand Up @@ -22,6 +22,7 @@ stdenv.mkDerivation rec {
preConfigure = ''
echo "${buildMK}" > mk/build.mk
sed -i -e 's|-isysroot /Developer/SDKs/MacOSX10.5.sdk||' configure
export NIX_LDFLAGS="$NIX_LDFLAGS -rpath $out/lib/ghc-${version}"
'';

configureFlags = "--with-gcc=${stdenv.gcc}/bin/gcc";
Expand Down

11 comments on commit d64917a

@vcunat
Copy link
Member

@vcunat vcunat commented on d64917a Oct 30, 2013

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems to cause many build problems on Hydra. I see mainly i686-linux and darwin fail.

@peti
Copy link
Member Author

@peti peti commented on d64917a Oct 31, 2013

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hydra failed to compile GHC because the build machines ran out of memory (probably because they were compiling several versions of GHC at the same time due to my commit). GHC is a dependency for several hundred other builds. I've asked to please restart those GHC builds, but apparently no-one had time to do it yet. I can't do it, since I don't have an account on Hydra.

@vcunat
Copy link
Member

@vcunat vcunat commented on d64917a Oct 31, 2013

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm sorry, I was probably too tired when writing this. I did read the e-mail a few days before, etc.

@kosmikus
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@peti I don't completely understand where this is coming from, but I'm having trouble building my old environments now due to collision errors. In particular, I can no longer have the "haddock" package in an environment (I get a collision between the haddock binary from ghc and the one provided by the package; but what if I need the haddock library?). I also just now got a collision between "runhaskell" as provided by ghc and ghc-wrapper which is even more confusing to me.

@peti
Copy link
Member Author

@peti peti commented on d64917a Oct 31, 2013

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kosmikus, the behavior of the wrapper has changed. Versions prior to this commit included only a hand-picked subset of the wrapped packages' directory paths into $out -- mostly just the Haskell libraries. In the new version, every path that's part of "packages" ends up being in $out.

Like everything else, this change comes with pros and cons. One obvious disadvantage is that now some collisions occur, which didn't occur before, i.e. the "haddock" binary provided by ghc collides with the one from the haddock library. Furthermore, the ghc-wrapper that's propagated by "haskellPlatform" now collides with the ghc script that this wrapper provides.

The advantage of the new behavior is that all files provided by the library packages are visible in the derivation. If you'd wrap "ghcMod", then the Haskell library as well as "bin/ghc-mod", "share/emacs/site-lisp", and "share/doc/ghc-mod-3.1.3/html" are all included in $out! The previous version of the wrapper wouldn't have picked those files up (unless someone would have manually extended the script to do so).

One might argue that the old wrapper ran into the same collisions as the new one, it just didn't report them. If you'd use the old wrapper to wrap "haddock", then there'd still be a collision between the two versions of the binary, but the old wrapper would just silently choose the binary from GHC without saying anything.

Clearly, we need a solution for this "haddock" and "haskellPlatform" issue. Here are possible solutions I can think of:

  1. Remove bin/haddock from $ghc to avoid the collision with the haddock library.

  2. Allow people to instantiate the wrapper passing a flag "allowCollisions = true", and have the wrapper pick up files in the order in which they're specified in the packages list so that the last occurance wins. This would change the interface of the ghcWithPackages function, though, which currently doesn't accept a `dictionary argument.

  3. Remove 'ghc' from 'propagatedBuildInputs' in haskellPlatform.

  4. Modify ghcWithPackages to filter ghc-wrapper from the list of to-be-wrapper packages.

What do you think?

@peti
Copy link
Member Author

@peti peti commented on d64917a Nov 2, 2013

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Referencing #1161 from here.

@aristidb
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think for now it would make sense to at least exclude the binaries that cause conflicts. I for one would like to keep haskellPlatform in my environment definition without having to specify every single participant package other than GHC.

It might also make sense to have an overridable flag in haskellPlatform whether to include/export GHC.

@peti
Copy link
Member Author

@peti peti commented on d64917a Nov 2, 2013

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With regard to Haskell Platform, we have two separate use cases:

  1. Some users want to install Haskell Platform X.Y.Z. That is GHC plus a well-defined set of libraries, and all those packages come in very specific versions.

  2. Other users want to install a GHC version of their choice that knows about all those libraries that are in Haskell Platform, but they don't care whether the respective version numbers match the HP standard to the last digit. This use case treats HP more like a sort of convenient "default environment" which is likely to include all basic building blocks needed by the discerning Haskell hacker.

Now, the current haskellPackages_ghcXYZ.haskellPlatform attribute serves use case (1). These expressions conform (mostly) the standard, and they include exactly those parts that HP say they would -- including the compiler.

For use case (2), however, these attributes are inappropriate. Suppose I wanted to instantiate GHC 7.0.4 together with the libraries from the latest HP standard. Then I'd write:

with haskellPackages_ghc704;
ghcWithPackages (self: [haskellPlatform_2013_2_0_0])

But this is clearly non-sense, because HP 2013.2.0.0 includes GHC 7.6.3, so what does it mean to instantiate it with GHC 7.0.4? Indeed, the ghcWithPackages wrapper won't instantiate this derivation precisely because the two GHC instances collide.

So what we ought to do in my humble opinion is to provide two separate solutions for those two use cases. To serve use case (2), we should have simple lists in haskellPackages_ghcXYZ that simply gather the parts of the respective HP version for easy re-use:

# haskell-packages.nix
{ ...
  haskellPlatform_2009_2_0_2 = with self; [ cgi, editline, fgl, ... ];
  haskellPlatform_2010_1_0_0 = with self; [ cgi, editline, fgl, ... ];
  ...
  haskellPlatform = self.haskellPlatform_X_Y_Z; # compiler-dependent
  ...
}

For use case (1), however, there should be appropriate packages in the top-level:

# all-packages.nix
{ ...
  haskellPlatform_2009_2_0_2 = with haskellPackages_ghc6104;
    ghcWithPackages (self: [haskellPlatform_2009_2_0_2]);
  haskellPlatform_2010_1_0_0 = with haskellPackages_ghc6123;
    ghcWithPackages (self: [haskellPlatform_2010_1_0_0]);
  ...
  haskellPlatform = haskellPlatform_2013_2_0_0;
  ...
}

This re-factoring would come with a (significant) additional benefit: The Haskell Platform derivations we currently have don't interact nicely with user-land cabal-install, because ghc-wrapper does not. The ghcWithPackages wrapper coexists with cabal-install just fine, though, which would make this setup more robust to use for beginners.

@kosmikus
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll have to look at all of this in more detail soon. For now, I've reverted to the pre-stdenv-updates version for all my systems, as the changes turned out to completely break my workflow.

@peti
Copy link
Member Author

@peti peti commented on d64917a Nov 4, 2013

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Commit 0e831bd adds an adapted version the old wrapper code again. This should allow you to use ghcWithPackagesOld to get the previous behavior.

@peti
Copy link
Member Author

@peti peti commented on d64917a Nov 7, 2013

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kosmikus, @aristidb, and everyone else who has reverted back to ghcWithPackagesOld, could you please check that the following re-implementation of ghcWithPackagesOld works fine for you and let me know what you find? The patch is very cheap to build -- testing it is not much effort.

Please sign in to comment.