Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Custom GHC build #4567

Closed
hsyl20 opened this issue Feb 5, 2019 · 19 comments

Comments

@hsyl20
Copy link
Contributor

commented Feb 5, 2019

I have some code that depends on a custom patched GHC. Currently my options to allow other people to build this code are:

  1. Create a GHC bindist. Make it available somewhere online. Add this kind of configuration into stack.yaml:
setup-info:
  ghc:
     linux64-custom-bugfix-tinfo6:
        8.6.3:
           url: "http://myserver/downloads/ghc-8.6.3-x86_64-bugfix-linux.tar.xz"

compiler: ghc-8.6.3
ghc-variant: bugfix
allow-newer: true
  1. Give instructions to user on how to build the custom GHC and tell them to use the following Stack configuration updated for their system:
extra-path:
- /home/login/repo/ghc/_build/stage1/bin

system-ghc: true
compiler: ghc-8.7.20190131
allow-newer: true

Both options are not satisfying.

What I would like instead is to be able to specify in stack configuration:

compiler:
   git: https://github.com/me/ghc.git
   commit: 4b00a96cee23ba0bf658834709703368b02f7e30
   flavour: devel2

This would:

  1. download GHC from git (similarly to an extra-dep).
  2. build it using: ./hadrian/build.stack.sh -c -j --flavour=devel2 binary-dist
  3. install the resulting bindist into: STACK_ROOT/programs/ARCH/ghc-4b00a96cee23ba0bf658834709703368b02f7e30-devel2/
  4. use this GHC as a compiler
@dbaynard

This comment has been minimized.

Copy link
Contributor

commented Feb 5, 2019

Hi @hsyl20, thanks for the suggestion.

Both options are not satisfying.

I'd like to know the pros and cons of each of the options — they aren't obvious to me.

Would you be interesting in contributing and maintaining a solution?

@borsboom

This comment has been minimized.

Copy link
Contributor

commented Feb 6, 2019

One thing to keep in mind is that building GHC from source requires a lot extra stuff installed on the developer's machine, so it wouldn't work "out of the box". We've considered supporting building GHC from source in the past, but decided against it for this reason.

@hsyl20

This comment has been minimized.

Copy link
Contributor Author

commented Feb 6, 2019

@dbaynard

Solution 1:

  • Pro: users of my package only have to type "stack build"
  • Con: I need to build and to provide bindists for different architectures/systems which requires a lot of time and access to various platforms

Solution 2:

  • Pro: I don't have to provide any GHC bindist
  • Con: users of my package have to build GHC themselves and to modify stack.yaml accordingly. We could script it but it is still worst than just typing "stack build"

I have been trying to implement the feature, the code is in this branch: https://github.com/hsyl20/stack/tree/hsyl20-compiler-build The test is running but GHC takes ages to build on my station so I'll get the result tomorrow.

@borsboom
With Hadrian it may be easier than before. Hadrian itself can be built via Stack and it could install some of the extra stuff (happy, alex, etc.) via Stack too. In any case, we should tag the feature as experimental.

@hsyl20

This comment has been minimized.

Copy link
Contributor Author

commented Feb 7, 2019

Some progress report: with my branch we can now use the following config:

compiler-build:
   #git: https://gitlab.haskell.org/ghc/ghc.git
   git: /home/hsyl20/repo/ghc/master
   commit: 23b8cb5f98ff4d1cfc641a28743ab0b881d3b52e
   flavour: quickest

compiler: ghc-git-23b8cb5f98ff4d1cfc641a28743ab0b881d3b52e-quickest
  • we need a GHC which can create an installable bindist with Hadrian (i.e. with this MR merged).

  • stack setup correctly builds and installs GHC from source into STACK_ROOT/programs/ARCH/ghc-git-23b8cb5f98ff4d1cfc641a28743ab0b881d3b52e-quickest

  • stack path correctly reports the compiler-* paths

  • stack shouldn't use the version reported by the built GHC in some paths as it contains the building date (e.g. "8.7.20190207") which is meaningless. It should use commit+flavour instead. We should probably explicitly set the ProjectVersion (PACKAGE_VERSION for ./configure) of GHC when we build it to return something sensible too...

  • building a package generated with "stack new" and modified to use the feature works

  • trying to build a package that has "ghc" package as dependency fails with:

[debug] Constructing the build plan
No package revision found for: PackageName "ghc"
CallStack (from HasCallStack):
  error, called at src/Stack/Build/ConstructPlan.hs:428:42 in stack-1.10.0-4H9uzszpUTZCuvWV6hOhdW:Stack.Build.ConstructPlan

Any help appreciated on this last one.

@dbaynard

This comment has been minimized.

Copy link
Contributor

commented Feb 7, 2019

Any help appreciated on this last one.

Stack bug. See #4573. It should be merged today — just waiting to see if anybody else comments. Merged (to master).

@dbaynard

This comment has been minimized.

Copy link
Contributor

commented Feb 7, 2019

Whoops, accidentally ticked one of the boxes — undone it now.

The rest of your results looks pretty impressive.

The test is running but GHC takes ages to build on my station so I'll get the result tomorrow.

This is surely a difficulty for you, though. Will your users not mind having to recompile GHC?

@hsyl20

This comment has been minimized.

Copy link
Contributor Author

commented Feb 7, 2019

Stack bug. See #4573. It should be merged today — just waiting to see if anybody else comments.

Nice! I'm glad I have reported it here before investing time into fixing it

This is surely a difficulty for you, though. Will your users not mind having to recompile GHC?

It would be a temporary measure until the changes are integrated into a released GHC (if ever). I try to get them merged but it takes quite some time (e.g. I'm trying to get this MR merged for a month now).

@hsyl20

This comment has been minimized.

Copy link
Contributor Author

commented Feb 8, 2019

It almost works but I encounter a new issue similar to what is presented here: https://www.snoyman.com/blog/2019/01/mismatched-global-packages

The GHC HEAD bindist we build is bundled with Cabal-2.5.0.0 which isn't released on Hackage... Stack reports:

[debug] Ignoring package Cabal due to wanting version 2.4.1.0 instead of 2.5.0.0

However, Cabal-2.4.1.0 (the latest Cabal release on Hackage) can't be built against GHC HEAD.

If I add "Cabal-2.5.0.0" as an extra-dep, it can't be found on Hackage (obviously). Is there a way to make Stack try to use packages bundled with GHC?


Otherwise, a workaround could be to add extra-deps like this:

- git:  /home/hsyl20/repo/ghc/ghc
  commit: 2ddbfdd0b621c4f9cde9a4517d2831f52b506201 #same commit ID than for the compiler
  subdirs:
    - libraries/Cabal/Cabal/
    - libraries/.... # infer the actual list of subdirs and their paths somehow...

But it doesn't work because git archive used by Pantry doesn't archive files from submodules...
My last patch adds support for repos with submodules to Pantry, so it works now.

It also works if we try to add a dependency on the actual submodule repo/commit:

- git: /home/hsyl20/repo/ghc/packages/Cabal.git
  commit: 97484d8e46f3c542523ef5daf5470540a4d66cb6 # commit ID of the submodule!
  subdirs:
     - Cabal

If we could make Pantry use git archive-all or something similar (like removing submodules while keeping files before calling git archive), that would alleviate this issue (and it could be useful to others too). Done.

@qrilka

This comment has been minimized.

Copy link
Contributor

commented Feb 8, 2019

@hsyl20 sorry for getting into the middle of this conversation but what package do you compile so you get a warning about some package wanting version 2.4.1.0 of Cabal? I guess you could try bumping that dependency bound
As for a custom Cabal version Stack currently uses only the one which gets shipped with GHC, see #4488

@hsyl20

This comment has been minimized.

Copy link
Contributor Author

commented Feb 8, 2019

@qrilka It seems to be from a build dependency of cabal-doctest: Cabal <2.5 (used to get some CPP #define from Cabal package). I've tried:

  • allow-newer:true: still try to build Cabal 2.4.1.0
  • remove the upper bound on Cabal: same
  • remove Cabal build dependency: fails because of undefined CPP

I'm not sure it related to my patches as it is reproducible without using the new feature (edit: confirmed that it fails with Stack head too):

> stack new test
> cd test
> stack install cabal-doctest -v
... Ignoring package Cabal due to wanting version 2.4.1.0 instead of 2.4.0.1
... [Tries to build Cabal 2.4.1.0]

Or:

> stack unpack cabal-doctest
> cd cabal-doctest-1.0.6
> stack init
> stack build -v
... Ignoring package Cabal due to wanting version 2.4.1.0 instead of 2.4.0.1
@qrilka

This comment has been minimized.

Copy link
Contributor

commented Feb 8, 2019

Actually I think I have confused myself here (and maybe you too?) - #4488 talks about using Cabal to build projects for that purpose we use Cabal shipped with GHC only and if Cabal is a dependency of another package then Stackage snapshot could override it but then you pay extra price of compiling Cabal :(
If you have Cabal-2.5.0.0 in extra-deps it should override Cabal-2.4.1.0 from LTS-13 (I suppose that's the snapshot you're using)

@dbaynard

This comment has been minimized.

Copy link
Contributor

commented Feb 9, 2019

Eek, Cabal is complicated. See #4494 for further info.

@hsyl20 hsyl20 referenced this issue Feb 11, 2019
2 of 2 tasks complete
@hsyl20

This comment has been minimized.

Copy link
Contributor Author

commented Feb 12, 2019

If you have Cabal-2.5.0.0 in extra-deps it should override Cabal-2.4.1.0 from LTS-13 (I suppose that's the snapshot you're using)

I can't specify Cabal-2.5.0.0 in extra-deps directly as it isn't released but I can specify it via its Git repository. It may rebuild a Cabal that has already been built when building GHC but we have no easy way to avoid it. Also it can be useful to pass different options to build Cabal, so we just have to document this. I opened #4581 to add support for Git submodules so that GHC's repository can be used to get a working Cabal for the built GHC more easily.

@qrilka

This comment has been minimized.

Copy link
Contributor

commented Feb 12, 2019

Sorry @hsyl20 for talking only about extra-deps, what I meant is that you could have Cabal-2.5.0.0 as a project package (as e.g. https://github.com/ghc/ghc/blob/master/hadrian/stack.yaml) or an extra-dep (using any method - Hackage, local directory or git) and then it should be used by Stack for any project depending on Cabal. But for building packages only Cabal shipped with GHC will be used. But maybe we'll add some way in #4488 to override Cabal for that purpose too.

@hsyl20

This comment has been minimized.

Copy link
Contributor Author

commented Feb 12, 2019

But for building packages only Cabal shipped with GHC will be used.

According to #4494 (see here), Custom setups are built with Cabal from the snapshot (or overloaded with extra-deps), not with Cabal shipped with GHC. That seems to explain what's happening when I try to build cabal-doctest.

It might be worth adding a way to specify an extra-dep to a boot package:

extra-deps:
  - boot: Cabal

So that Cabal shipped with GHC is used instead of the snapshot one even for Custom builds.

@qrilka

This comment has been minimized.

Copy link
Contributor

commented Feb 12, 2019

Oh, custom setups, sorry, completely forgot about those

@snoyberg

This comment has been minimized.

Copy link
Contributor

commented Mar 25, 2019

Issue #4637 also asks for the ability to build GHC from source. I'm leaning towards wishlisting both of these issues until someone steps up to contribute code to make this happen.

@hsyl20

This comment has been minimized.

Copy link
Contributor Author

commented Mar 25, 2019

@snoyberg I have polished my patch and submitted PR #4655

@hsyl20 hsyl20 referenced this issue Mar 29, 2019
3 of 3 tasks complete

snoyberg added a commit that referenced this issue Mar 30, 2019

snoyberg added a commit that referenced this issue Apr 1, 2019

snoyberg added a commit that referenced this issue Apr 1, 2019

snoyberg added a commit that referenced this issue Apr 1, 2019

Merge pull request #4671 from commercialhaskell/drop-packages-in-stac…
…k-yaml

Add drop-packages to stack.yaml #4567
@snoyberg

This comment has been minimized.

Copy link
Contributor

commented Apr 13, 2019

Implemented in #4655, closing. Thank you for the contribution!

@snoyberg snoyberg closed this Apr 13, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.