Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build GHC from source (#4567) #4655

Merged
merged 3 commits into from Apr 10, 2019

Conversation

@hsyl20
Copy link
Contributor

@hsyl20 hsyl20 commented Mar 25, 2019

This patch adds the experimental support for building GHC from source. We can now specify a specific GHC to build from source in stack.yaml with the following syntax:

compiler-build:
   git: GHC_GIT_REPO
   commit: COMMIT
   flavour: FLAVOUR

In the setup phase Stack uses Hadrian to build a GHC binary distribution
with the specified flavour and installs it into
STACK_ROOT/programs/ARCH/ghc-git-COMMIT-FLAVOUR.

The built compiler can be used with the following configuration:

compiler: ghc-git-COMMIT-FLAVOUR

As GHC may rely on unreleased version of the global packages (e.g.
Cabal), you may have to add some extra-deps as follows:

extra-deps:
- git: GHC_GIT_REPO
  commit: COMMIT
  subdirs:
    - libraries/Cabal/Cabal
    - libraries/...
  • Any changes that could be relevant to users have been recorded in the ChangeLog.md
  • The documentation has been updated, if necessary.
  • A test has been added but it takes a long time to run as it builds GHC. [Also note that currently GHC head builds non-installable bindists, cf GHC#16498]
@hsyl20 hsyl20 mentioned this pull request Mar 25, 2019
@hsyl20 hsyl20 force-pushed the hsyl20-compiler-build branch 3 times, most recently from a7171d4 to 3353093 Mar 26, 2019
@snoyberg
Copy link
Contributor

@snoyberg snoyberg commented Mar 28, 2019

Thanks! A few initial comments

  • I see that there's still a CI failure, can you address that?
  • How do you specify which version of GHC is used for bootstrapping?
  • Can you clarify a bit more about the global packages? Stack tries to assume that global packages are not the same as something on Hackage

@hsyl20 hsyl20 force-pushed the hsyl20-compiler-build branch 3 times, most recently from 295e270 to a79c607 Mar 29, 2019
@hsyl20
Copy link
Contributor Author

@hsyl20 hsyl20 commented Mar 29, 2019

* I see that there's still a CI failure, can you address that?

I'm trying to fix it now. However I'm not sure I've done the correct thing in src/Stack/Types/BuildPlan.hs. I've blindly updated the version string but I'm not sure about it.

* How do you specify which version of GHC is used for bootstrapping?

I have updated the documentation. But to answer here: the bootstrapping compiler is configured via hadrian/stack.yaml in GHC source tree. Hence the commit id also "hashes" the bootstrap compiler used, which is very good for reproducible builds.

* Can you clarify a bit more about the global packages? Stack tries to assume that global packages are not the same as something on Hackage

I have tried to improve the documentation on this topic but I don't really understand " Stack tries to assume that global packages are not the same as something on Hackage". The issue we had in #4567 (cf #4567 (comment)) is that Stack was trying to build an older Cabal against a newer GHC while the only working Cabal was still unreleased.

@hsyl20 hsyl20 force-pushed the hsyl20-compiler-build branch from a79c607 to 5263e0d Mar 29, 2019
@snoyberg
Copy link
Contributor

@snoyberg snoyberg commented Mar 31, 2019

I'm trying to fix it now. However I'm not sure I've done the correct thing in src/Stack/Types/BuildPlan.hs. I've blindly updated the version string but I'm not sure about it.

Thanks! I'll review the code in more detail soon.

I have updated the documentation. But to answer here: the bootstrapping compiler is configured via hadrian/stack.yaml in GHC source tree. Hence the commit id also "hashes" the bootstrap compiler used, which is very good for reproducible builds.

That's even better than I was hoping for!

I have tried to improve the documentation on this topic but I don't really understand " Stack tries to assume that global packages are not the same as something on Hackage". The issue we had in #4567 (cf #4567 (comment)) is that Stack was trying to build an older Cabal against a newer GHC while the only working Cabal was still unreleased.

I think the issue here is that the Stackage snapshots all specify an override of the Cabal library, which forces Stack to ignore the global Cabal version for custom Setup.hs files. We may be moving away for that for other reasons. There are more details at commercialhaskell/stackage#4425.

In #4671 I've added support for a drop-packages configuration, which will drop the Cabal library override and hopefully make this use case work more easily.

Copy link
Contributor

@snoyberg snoyberg left a comment

This looks really good.

doc/yaml_configuration.md Outdated Show resolved Hide resolved
doc/yaml_configuration.md Outdated Show resolved Hide resolved
doc/yaml_configuration.md Outdated Show resolved Hide resolved
doc/yaml_configuration.md Show resolved Hide resolved
src/Stack/Script.hs Show resolved Hide resolved
src/Stack/Setup.hs Outdated Show resolved Hide resolved
src/Stack/SetupCmd.hs Outdated Show resolved Hide resolved
src/Stack/Types/BuildPlan.hs Outdated Show resolved Hide resolved
src/Stack/Types/CompilerGitBuild.hs Outdated Show resolved Hide resolved
subs/pantry/src/Pantry/Types.hs Show resolved Hide resolved
@hsyl20 hsyl20 force-pushed the hsyl20-compiler-build branch 3 times, most recently from c42870d to e93efc1 Apr 9, 2019
@hsyl20
Copy link
Contributor Author

@hsyl20 hsyl20 commented Apr 9, 2019

Thanks @snoyberg for the review! I've updated the PR accordingly.

@hsyl20 hsyl20 force-pushed the hsyl20-compiler-build branch from e93efc1 to b51cf31 Apr 9, 2019

-- RIO.Process doesn't wrap process' "shell".
-- Instead we use "proc" with the "sh" command
proc "sh" hadrianArgs runProcess_
Copy link
Contributor

@snoyberg snoyberg Apr 10, 2019

I don't think this will work on Windows, since we're trying to run a batch file, not a shell script. What about instead:

proc hadrianCmd hadrianArgs runProcess_

And removing the hadrianCmd from the args above.

Copy link
Contributor Author

@hsyl20 hsyl20 Apr 10, 2019

IIRC it wasn't working this way on Linux: we would need shell but RIO only has proc AFAIK.

Copy link
Contributor

@snoyberg snoyberg Apr 10, 2019

Is the shell script in the repo set to be executable?

Copy link
Contributor Author

@hsyl20 hsyl20 Apr 10, 2019

Yes it is. It fails with "Did not find executable at specified path: ./hadrian/build.stack.sh"

Copy link
Contributor

@snoyberg snoyberg Apr 10, 2019

Please see 2768bb9. This is a really confusing aspect of the process package, I only figured this out since I've been bitten by it before.

src/Stack/Setup.hs Outdated Show resolved Hide resolved
src/Stack/Setup/Installed.hs Show resolved Hide resolved
hsyl20 added 3 commits Apr 10, 2019
We can now specify a specific GHC to build from source in `stack.yaml`
with the following syntax:

```
compiler: ghc-git-COMMIT-FLAVOUR
```

In the setup phase Stack downloads the source from the specified Git
repository and commit ID, then it uses Hadrian to build a GHC binary
distribution with the specified flavour and it installs it into
STACK_ROOT/programs/ARCH/ghc-git-COMMIT-FLAVOUR.
@hsyl20 hsyl20 force-pushed the hsyl20-compiler-build branch from b51cf31 to 36d6e1b Apr 10, 2019
@snoyberg
Copy link
Contributor

@snoyberg snoyberg commented Apr 10, 2019

OK, the only thing remaining is the execution of the script on POSIX vs Windows. Everything else looks great. I'm going to futz with it a bit on my OS X and Windows machines and then merge to master. Thanks for enduring a thorough review process @hsyl20 and for the new feature!

@snoyberg snoyberg merged commit 36d6e1b into commercialhaskell:master Apr 10, 2019
8 checks passed
@hsyl20
Copy link
Contributor Author

@hsyl20 hsyl20 commented Apr 10, 2019

Thanks Michael for the review! My patch fixing GHC bindists has also been merged into GHC so the test passes (on Linux)!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

3 participants