Building a package with "src = ./." can fail if the "dist" directory has been set-up outside of Nix

[stepcut]

If I use cabal2nix to generate a default.nix for the current directory ./., and I have a dist directory hanging around, that directory gets included in the source. This later leads to build errors because nix builds with the -split-objs flag, but the GHC recompilation checker doesn't rebuild the .o files because the source has not changed since they were last built. This later results in it not finding the split .o files. While that could be considered a bug in GHC, it seems wrong to even include the dist directory in the first place.

It would be nice if cabal2nix automatically filtered some directories from the source using filterSource:

http://hackage.haskell.org/package/hspec-checkers-0.1.0/src/default.nix

Obvious choices would be .git, dist, and _darcs, etc, in the top-level of a project. Though in my personal projects I would also want to ignore _state. So, it would be nice if there was something like a .cabal2nix file where I could set my preferred boring files.

[bergey]

I think I've been working around this by running cabal clean a lot. I hadn't worked out why this was necessary, but I'd very much appreciate a better solution.

Would it make more sense for cabal.mkDerivation to call filterSource, though? I assume it's only needed for local packages (src = ./.), but repeating it in every local default.nix doesn't seem very good, either.

[stepcut]

I am not sure where filterSource should actually be called. There is a list of boring files that can almost always be ignored, and putting that potentially long list in every generated default.nix would be annoying. It also makes it hard to add other files later, because already generated .nix files would not be affected. There will definitely be times where a package will want to explicitly allow a normally ignored file/directory and times when additional files/directories should be ignored.

So modifying cabal.mkDerivation sounds good as long as is it easy to alter the list when you want different behavior.

One option might be to put the filterSource call in the .nix but the standard list of boring files could be imported from a centralized place.

[peti]

I don't think that this feature belongs into the default Cabal builder. IMHO, it would be fine to have an extended version of pkgs/build-support/cabal/default.nix -- say pkgs/build-support/cabal/interactive.nix -- that extends the default function with this functionality. One advantage of separating the two is that changes to the extended builder won't cause re-builds of all Haskell packages in Nixpkgs, etc.

[stepcut]

I think I'd still argue for cabal2nix generating a the filter list, but only when generating a default.nix for ./.. I like having the list of excluded files in the generated default.nix because then it is easier to add additional exclusions on a per-project basis. With the initial list being right in the file it is obvious that files are being excluded and easy to change the list of exclusions. As I suggested earlier, there could be some global boring file list that it imports as a starting point. Obviously, anytime that boring list is changed it could trigger a large number of rebuilds. However, that would only be rebuilds of things with src = ./., so that might not be too bad.

[stepcut]

Alternatively, what if cabal2nix used cabal sdist. That way it would more accurately reflect the results I will get when I upload the package to hackage?

[stepcut]

Seems cabbage uses sdist, https://github.com/acowley/cabbage/blob/master/Nix-notes.org

[stepcut]

actually, I think there should just be a general purpose cabalsdist that runs cabal sdist. Here is a quick prototype. There is a directory cabalsdist which contains two files.

default.nix:

{stdenv, pkgs }:
{pathname}:
stdenv.mkDerivation
{ name = "cabalsdist";
  builder = ./builder.sh;
  buildInputs = [ pkgs.haskellPackages.cabal-install ];
  pathname = pathname;
}

And builder.sh:

source $stdenv/setup

(cd $pathname ; cabal sdist --output-directory "$out")

Then you use callPackage to make cabalsdist available:

  packageOverrides = superDuper:
    let self = superDuper.pkgs;
  in
  {
    cabalsdist = superDuper.callPackage /Users/stepcut/.nixpkgs/cabalsdist {};
    haskellPackages = superDuper.haskellPackages.override rec {
    <snip>

And then in the default.nix you can use it as follows:

...
mkDerivation {
  pname = "happstack-server";
  version = "7.4.3";
  src = cabalsdist { pathname = ./.; };
...

[peti]

IMHO, it would be easiest if cabal2nix would generate an appropriate shell.nix expression that adds the necessary filter code when the src attribute points to the local disk. Then the default builder could remain unchanged, but we'd still get reliable builds for everyone without them having to edit the generated files. Which is pretty much what @stepcut suggested in #112 (comment).

[stepcut]

I would still suggest that the cabalsdist solution is even better. It ensures that only the files that will actually make it onto hackage are used. Which is super useful for development work. Additionally, it does not require any modifications to the default builder. (The builder.sh script I provided is for cabalsdist itself, but the actual building of the packages is unmodified).

[peti]

I agree that using sdist is nice. I'll have to experiment with that approach a little, though, to understand the implications better.

[chris-martin]

The lack of more granular control over the source path is a major pain point for me. Most projects I write nowadays include some script to edit the generated nix file to add a filterSource call to exclude dotfiles that trigger unnecessary rebuilds.

[Ptival]

This has become even more of a pain point now that I'm using lorri. If you naively use callCabal2nix to generate nix expressions for your local packages, the created package will include all of your files, and this may trigger lorri for unrelated file changes! I noticed my CPU at 100% a lot! :-)

The problem is not only with dist. For some development checkout of a Haskell project, I might have all sorts of garbage in the directory that should not make it in the derivation: dist, dist-newstyle, .stack-work, .git, etc.

For now, I guess I'll have to revert to this kind of ugly hack.

EDIT:
I found a way better solution by Profpatsch:
#428