Add ways to build hadrian using nix #473

angerman · 2017-11-17T13:17:40Z

This adds two new files to the hadrian directory

shell.nix sets up the build envrionment you need to build ghc
build-nix is a simple wrapper which invokes hadrian in the correct environment

Note: this patch was authored by @mpickering, however it ended up on phabricator due to the subtree as https://phabricator.haskell.org/D4207.

snowleopard · 2017-11-17T13:21:34Z

build-nix

@@ -0,0 +1,32 @@
+#!/usr/bin/env nix-shell
+#! nix-shell -i bash shell.nix


This file looks almost identical to build.stack.nix.sh -- what's the difference? Do we need both?

I see, it is quite similar. But the obvious difference being that this doesn't require stack nor cabal to be installed on your system and is more direct than using a stack.yaml file to specify the nix configuration.

I just copied this in my tree for now and would therefore be the second user of this approach. (would like to avoid using stack or cabal-install)

So, to use this script one will first have to somehow build Hadrian. This feels like an easy way to shoot oneself in the foot by using an outdated Hadrian binary.

It may be a good idea to separate the two concerns somehow: building Hadrian and building GHC, but this PR seems to only address the latter, but not the former, unless I'm missing something?

(I've never used nix, so excuse my naive comments!)

When you call this script, it enters the environment specified in shell.nix, which "provisions" among other things a local build of hadrian.

(and it has everything needed for building ghc, too)

snowleopard · 2017-11-17T13:25:10Z

Can we stick to the convention that shell scripts have .sh extension?

A general comment: I am concerned that we keep accumulating too many various build scripts, which can be confusing. Perhaps we should only keep the default build.sh and build.bat at the top level and create a separate directory called scripts for custom build scripts?

snowleopard · 2017-11-17T13:25:55Z

@mpickering @angerman Thanks for the PR and for bringing it here!

snowleopard · 2017-11-17T13:44:13Z

shell.nix

+        HUnit = nixpkgs.haskell.lib.dontCheck (self.callHackage "HUnit" "1.3.1.2" {});
+      }; };
+
+  hadrian = hadrianPackages.ghcWithPackages (ps: [ ps.hadrian ]);


I'm not familiar with nix, so perhaps this is the second part of the puzzle -- is this the nix-based build script for Hadrian?

It overrides the default haskell packageset by replacing hadrian, etc by the local ones, except for shake for which we specifically require 0.16 from hackage, and same with HUnit 1.3.1.2. Finally, line 19 uses this "new" package set to define a build environment that has a ghc + hadrian and all of its dependencies in it. This is then used along with many other things to provision the entire build environment needed to build ghc, in the rest of the file.

So, if the user edits hadrian/UserSettings.hs will the invokation of the build-nix script automatically rebuild Hadrian?

Yes. In fact, it will even do it when any file from the hadrian dir changes, but this is a quirk @mpickering told me about and we know how to fix it (it's quite simple, nothing fancy, just requires calling a function or two in the nix code). If @mpickering doesn't get around to doing it in the upcoming days I'll do it.

@alpmestan Note that in principle Hadrian should also be rebuilt when the in-tree Cabal library is modified. But this is a rather rare scenario.

That is also taken care of correctly but not evident from the script.

Essentially callCabal2nix creates a function which takes all the dependencies of hadrian and returns a recipe to construct hadrian using these dependencies. It then performs some magic to automatically fill in these arguments correctly and because we specify that we want to use the in-tree Cabal on a later line, it passes this version of Cabal as an argument. The key principle of nix is that when an argument changes, then you have to rebuild the result, so when the in-tree Cabal changes, then Cabal and hence hadrian will be rebuilt.

Aha, that's cool!

We got you halfway to using nix haven't we? :)

@alpmestan Yes, you have :) I've already heard a lot of good things about nix and I'm tempted!

snowleopard · 2017-11-17T14:10:21Z

/cc @duog who added the build.stack.nix.sh script in #430.

mpickering · 2017-11-17T15:18:26Z

There are two files.

The first shell.nix can be invoked directly by calling nix-shell shell.nix and its goal is to set up an environment in which you can build ghc. As such, it adds dependencies to the environment such as gmp, python3 and so on. One such dependency is hadrian itself, so the instructions about how to build hadrian are automatically calculated from hadrian.cabal. This happens in the call callCabal2nix which looks in the given directory for a cabal file and creates a nix expression based on that which is can use to build the library.

The second build-nix is a convenient wrapper around this shell.nix which automatically invokes hadrian in the correct environment.

The advantage of this script for nix users is that they don't have to use cabal or stack in order to manage their dependencies, they can just use the normal way they manage dependencies.

Hope that clears things up.

snowleopard · 2017-11-17T18:14:29Z

@mpickering Thanks, could you add this clarification as a comment on top of these scripts? I'm sure I'm not the only one who might be confused by all the different scripts we have. I'll try to add comments to other scripts too.

duog · 2017-11-17T18:57:55Z

@snowleopard: I added build.stack.nix.sh to allow build.stack.sh to not require nix.

snowleopard · 2017-11-17T23:41:23Z

I added build.stack.nix.sh to allow build.stack.sh to not require nix.

@duog Right. Do we need all build-nix, build.stack.sh and build.stack.nix.sh? Sounds a like there may be potential redundancy here...

@mpickering

This adds two new files to the hadrian directory shell.nix sets up the build envrionment you need to build ghc build-nix is a simple wrapper which invokes hadrian in the correct environment Note: this patch was authored by @mpickering, however it ended up on phabricator due to the subtree as https://phabricator.haskell.org/D4207.

mpickering · 2017-11-17T23:53:00Z

I improved the script this evening so that it doesn't rebuild hadrian so often and works with ghc802 as well.

I need to add some comments and non-functional changes tomorrow.

alpmestan · 2017-11-18T09:45:54Z

Perhaps we should only keep the default build.sh and build.bat at the top level and create a separate directory called scripts for custom build scripts?

Probably a good idea yes. With a short README that says "use this one if you want to build hadrian using X" for each specific script.

snowleopard · 2017-11-18T13:14:08Z

@alpmestan Indeed, that will work well. I've opened an issue to track this: #475.

duog · 2017-11-19T20:30:14Z

@snowleopard: Yes, it does seem there is redundancy. Unfortunately I don't understand nix, specifically the stack-nix integration, well enough to have an opinion on what should be done. Presumably the nix section in stack.yaml was added to help someone's workflow, we'd need to check with them.

mpickering · 2017-11-20T19:50:11Z

I added some comments now and renamed the file. Anything else left to do?

snowleopard · 2017-11-20T20:06:53Z

@mpickering Many thanks, that's great. Merged.

snowleopard reviewed Nov 17, 2017

View reviewed changes

mpickering force-pushed the feature/build-with-nix branch from 30c028d to a228f38 Compare November 17, 2017 23:51

snowleopard mentioned this pull request Nov 18, 2017

Move custom build scripts to a separate directory #475

Open

2 tasks

Comments and renamed build-nix

ce80f25

snowleopard merged commit 8fd6818 into snowleopard:master Nov 20, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add ways to build hadrian using nix #473

Add ways to build hadrian using nix #473

angerman commented Nov 17, 2017

snowleopard Nov 17, 2017

mpickering Nov 17, 2017

alpmestan Nov 17, 2017 •

edited

snowleopard Nov 17, 2017

snowleopard Nov 17, 2017

alpmestan Nov 17, 2017

alpmestan Nov 17, 2017

snowleopard Nov 17, 2017

snowleopard commented Nov 17, 2017

snowleopard commented Nov 17, 2017

snowleopard Nov 17, 2017

alpmestan Nov 17, 2017 •

edited

snowleopard Nov 17, 2017

alpmestan Nov 17, 2017 •

edited

snowleopard Nov 17, 2017

mpickering Nov 17, 2017

snowleopard Nov 17, 2017 •

edited

alpmestan Nov 17, 2017

snowleopard Nov 17, 2017

snowleopard commented Nov 17, 2017

mpickering commented Nov 17, 2017

snowleopard commented Nov 17, 2017

duog commented Nov 17, 2017

snowleopard commented Nov 17, 2017

mpickering commented Nov 17, 2017

alpmestan commented Nov 18, 2017

snowleopard commented Nov 18, 2017

duog commented Nov 19, 2017

mpickering commented Nov 20, 2017

snowleopard commented Nov 20, 2017

		@@ -0,0 +1,32 @@
		#!/usr/bin/env nix-shell
		#! nix-shell -i bash shell.nix

Add ways to build hadrian using nix #473

Add ways to build hadrian using nix #473

Conversation

angerman commented Nov 17, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alpmestan Nov 17, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

snowleopard commented Nov 17, 2017

snowleopard commented Nov 17, 2017

Choose a reason for hiding this comment

alpmestan Nov 17, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alpmestan Nov 17, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

snowleopard Nov 17, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

snowleopard commented Nov 17, 2017

mpickering commented Nov 17, 2017

snowleopard commented Nov 17, 2017

duog commented Nov 17, 2017

snowleopard commented Nov 17, 2017

mpickering commented Nov 17, 2017

alpmestan commented Nov 18, 2017

snowleopard commented Nov 18, 2017

duog commented Nov 19, 2017

mpickering commented Nov 20, 2017

snowleopard commented Nov 20, 2017

alpmestan Nov 17, 2017 •

edited

alpmestan Nov 17, 2017 •

edited

alpmestan Nov 17, 2017 •

edited

snowleopard Nov 17, 2017 •

edited