Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build GHC from source (#4567) #4655

Merged
merged 3 commits into from Apr 10, 2019

Conversation

Projects
None yet
3 participants
@hsyl20
Copy link
Contributor

commented Mar 25, 2019

This patch adds the experimental support for building GHC from source. We can now specify a specific GHC to build from source in stack.yaml with the following syntax:

compiler-build:
   git: GHC_GIT_REPO
   commit: COMMIT
   flavour: FLAVOUR

In the setup phase Stack uses Hadrian to build a GHC binary distribution
with the specified flavour and installs it into
STACK_ROOT/programs/ARCH/ghc-git-COMMIT-FLAVOUR.

The built compiler can be used with the following configuration:

compiler: ghc-git-COMMIT-FLAVOUR

As GHC may rely on unreleased version of the global packages (e.g.
Cabal), you may have to add some extra-deps as follows:

extra-deps:
- git: GHC_GIT_REPO
  commit: COMMIT
  subdirs:
    - libraries/Cabal/Cabal
    - libraries/...
  • Any changes that could be relevant to users have been recorded in the ChangeLog.md
  • The documentation has been updated, if necessary.
  • A test has been added but it takes a long time to run as it builds GHC. [Also note that currently GHC head builds non-installable bindists, cf GHC#16498]

@hsyl20 hsyl20 referenced this pull request Mar 25, 2019

Closed

Custom GHC build #4567

@hsyl20 hsyl20 force-pushed the hsyl20:hsyl20-compiler-build branch 3 times, most recently from a7171d4 to 3353093 Mar 26, 2019

@snoyberg

This comment has been minimized.

Copy link
Contributor

commented Mar 28, 2019

Thanks! A few initial comments

  • I see that there's still a CI failure, can you address that?
  • How do you specify which version of GHC is used for bootstrapping?
  • Can you clarify a bit more about the global packages? Stack tries to assume that global packages are not the same as something on Hackage

@hsyl20 hsyl20 force-pushed the hsyl20:hsyl20-compiler-build branch 3 times, most recently from 295e270 to a79c607 Mar 29, 2019

@hsyl20

This comment has been minimized.

Copy link
Contributor Author

commented Mar 29, 2019

* I see that there's still a CI failure, can you address that?

I'm trying to fix it now. However I'm not sure I've done the correct thing in src/Stack/Types/BuildPlan.hs. I've blindly updated the version string but I'm not sure about it.

* How do you specify which version of GHC is used for bootstrapping?

I have updated the documentation. But to answer here: the bootstrapping compiler is configured via hadrian/stack.yaml in GHC source tree. Hence the commit id also "hashes" the bootstrap compiler used, which is very good for reproducible builds.

* Can you clarify a bit more about the global packages? Stack tries to assume that global packages are not the same as something on Hackage

I have tried to improve the documentation on this topic but I don't really understand " Stack tries to assume that global packages are not the same as something on Hackage". The issue we had in #4567 (cf #4567 (comment)) is that Stack was trying to build an older Cabal against a newer GHC while the only working Cabal was still unreleased.

@hsyl20 hsyl20 force-pushed the hsyl20:hsyl20-compiler-build branch from a79c607 to 5263e0d Mar 29, 2019

@snoyberg

This comment has been minimized.

Copy link
Contributor

commented Mar 31, 2019

I'm trying to fix it now. However I'm not sure I've done the correct thing in src/Stack/Types/BuildPlan.hs. I've blindly updated the version string but I'm not sure about it.

Thanks! I'll review the code in more detail soon.

I have updated the documentation. But to answer here: the bootstrapping compiler is configured via hadrian/stack.yaml in GHC source tree. Hence the commit id also "hashes" the bootstrap compiler used, which is very good for reproducible builds.

That's even better than I was hoping for!

I have tried to improve the documentation on this topic but I don't really understand " Stack tries to assume that global packages are not the same as something on Hackage". The issue we had in #4567 (cf #4567 (comment)) is that Stack was trying to build an older Cabal against a newer GHC while the only working Cabal was still unreleased.

I think the issue here is that the Stackage snapshots all specify an override of the Cabal library, which forces Stack to ignore the global Cabal version for custom Setup.hs files. We may be moving away for that for other reasons. There are more details at commercialhaskell/stackage#4425.

In #4671 I've added support for a drop-packages configuration, which will drop the Cabal library override and hopefully make this use case work more easily.

@snoyberg
Copy link
Contributor

left a comment

This looks really good.

Show resolved Hide resolved doc/yaml_configuration.md Outdated
Show resolved Hide resolved doc/yaml_configuration.md Outdated
Show resolved Hide resolved doc/yaml_configuration.md Outdated
Show resolved Hide resolved doc/yaml_configuration.md
Show resolved Hide resolved src/Stack/Script.hs
Show resolved Hide resolved src/Stack/Setup.hs Outdated
Show resolved Hide resolved src/Stack/SetupCmd.hs Outdated
Show resolved Hide resolved src/Stack/Types/BuildPlan.hs Outdated
Show resolved Hide resolved src/Stack/Types/CompilerGitBuild.hs Outdated
Show resolved Hide resolved subs/pantry/src/Pantry/Types.hs

@hsyl20 hsyl20 force-pushed the hsyl20:hsyl20-compiler-build branch 3 times, most recently from c42870d to e93efc1 Apr 9, 2019

@hsyl20

This comment has been minimized.

Copy link
Contributor Author

commented Apr 9, 2019

Thanks @snoyberg for the review! I've updated the PR accordingly.

@hsyl20 hsyl20 force-pushed the hsyl20:hsyl20-compiler-build branch from e93efc1 to b51cf31 Apr 9, 2019


-- RIO.Process doesn't wrap process' "shell".
-- Instead we use "proc" with the "sh" command
proc "sh" hadrianArgs runProcess_

This comment has been minimized.

Copy link
@snoyberg

snoyberg Apr 10, 2019

Contributor

I don't think this will work on Windows, since we're trying to run a batch file, not a shell script. What about instead:

proc hadrianCmd hadrianArgs runProcess_

And removing the hadrianCmd from the args above.

This comment has been minimized.

Copy link
@hsyl20

hsyl20 Apr 10, 2019

Author Contributor

IIRC it wasn't working this way on Linux: we would need shell but RIO only has proc AFAIK.

This comment has been minimized.

Copy link
@snoyberg

snoyberg Apr 10, 2019

Contributor

Is the shell script in the repo set to be executable?

This comment has been minimized.

Copy link
@hsyl20

hsyl20 Apr 10, 2019

Author Contributor

Yes it is. It fails with "Did not find executable at specified path: ./hadrian/build.stack.sh"

This comment has been minimized.

Copy link
@snoyberg

snoyberg Apr 10, 2019

Contributor

Please see 2768bb9. This is a really confusing aspect of the process package, I only figured this out since I've been bitten by it before.

Show resolved Hide resolved src/Stack/Setup.hs Outdated
Show resolved Hide resolved src/Stack/Setup/Installed.hs

snoyberg added a commit that referenced this pull request Apr 10, 2019

hsyl20 added some commits Mar 29, 2019

Support building GHC from source
We can now specify a specific GHC to build from source in `stack.yaml`
with the following syntax:

```
compiler: ghc-git-COMMIT-FLAVOUR
```

In the setup phase Stack downloads the source from the specified Git
repository and commit ID, then it uses Hadrian to build a GHC binary
distribution with the specified flavour and it installs it into
STACK_ROOT/programs/ARCH/ghc-git-COMMIT-FLAVOUR.

@hsyl20 hsyl20 force-pushed the hsyl20:hsyl20-compiler-build branch from b51cf31 to 36d6e1b Apr 10, 2019

@snoyberg

This comment has been minimized.

Copy link
Contributor

commented Apr 10, 2019

OK, the only thing remaining is the execution of the script on POSIX vs Windows. Everything else looks great. I'm going to futz with it a bit on my OS X and Windows machines and then merge to master. Thanks for enduring a thorough review process @hsyl20 and for the new feature!

@snoyberg snoyberg merged commit 36d6e1b into commercialhaskell:master Apr 10, 2019

8 checks passed

commercialhaskell.stack Build #20190410.7 succeeded
Details
commercialhaskell.stack (Linux nightly) Linux nightly succeeded
Details
commercialhaskell.stack (Linux pedantic) Linux pedantic succeeded
Details
commercialhaskell.stack (Linux stack-def) Linux stack-def succeeded
Details
commercialhaskell.stack (Linux stack-lts-12) Linux stack-lts-12 succeeded
Details
commercialhaskell.stack (Linux style) Linux style succeeded
Details
commercialhaskell.stack (Windows stack) Windows stack succeeded
Details
commercialhaskell.stack (macOS stack-def) macOS stack-def succeeded
Details
@hsyl20

This comment has been minimized.

Copy link
Contributor Author

commented Apr 10, 2019

Thanks Michael for the review! My patch fixing GHC bindists has also been merged into GHC so the test passes (on Linux)!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.