haskell infra: Fix cross compilation to work with new system #26799

Ericson2314 · 2017-06-23T19:47:51Z

Motivation for this change

This is a re-doing of the Haskell support for cross compilation to fit in with my other cross refactoring. As a cheat sheet, with the new system:

Getting a cross compiler: haskell.packages.ghc{HEAD,821}.ghc, or buildPackages.haskell.compiler.ghc{HEAD,821}.ghc
Getting a cross-compiled package: haskell.packages.ghc{HEAD,821}.fooBarBaz

There's other PRs that need to be merged first, but I am opening this now so people can get a flavor for what's to come.

Fixes #21874

Things to do

Optional changes

~~Get rid of --with flags.~~ Actually, the --with flags are needed for GHC's own dynamic linker, long story short.
Add tests (for both Linux and Darwin) in pkgs/top-level/release-cross.nix. Linux should work right now, and darwin Later. Done for GHC 8.2.2 and GHC HEAD from x86_64 linux to 32-bit and 64-bit arm.
Update to a newer GHC commit with @bgamari's and @angerman's fixes (thanks!) so this actually builds. GHC HEAD is good to go, but GHC 822 needs one more patch to land to I have a more permanent URL: https://phabricator.haskell.org/D4287

Misc simple things

Update other GHCs likewise
Use configurePlatforms to control passing of --build, --host, and --target.
A few more misc changes to utilities like hackage2nix.
Break history into more separate commits?
targetPrefix is bound on GHC, so we don't need to recompute it all over the place.
Do make sure all crossPrefix or prefix, from earlier versions, are gone

CC @expipiplus1 @dtzWill @dhess

mention-bot · 2017-06-23T19:47:55Z

@Ericson2314, thanks for your PR! By analyzing the history of the files in this pull request, we identified @peti, @kosmikus and @pikajude to be potential reviewers.

shlevy

In addition to in-line comments, any way we can reduce duplication here?

shlevy · 2017-06-24T00:43:37Z

pkgs/development/compilers/ghc/8.2.1.nix

+  #
+  # We generally want the latest llvm package set, which would normally be
+  # `llvmPackages` on most platforms. But on Darwin, the default is the version
+  # released with OSX, so we force 3.9, which is the correct version at this


nixpkgs convention is to override versions like this at the callPackage site, not here.

Now overridden in pkgs/top-level/haskell-packags,nix.

shlevy · 2017-06-24T00:43:58Z

pkgs/development/compilers/ghc/8.2.1.nix

-
-  # If enabled GHC will be build with the GPL-free but slower integer-simple
+, ncurses, gmp, libffi
+, __targetPackages


Can you document this and other non-package non-obvious arguments?

This is technically something bound in all-packages, rather than a ghc-specific "configuration argument", but sure. Do note that in #26805 I add __targetInputs and __propagatedTargetInputs so there is no need for this parameter.

Now documented in manual

shlevy · 2017-06-24T00:44:31Z

pkgs/development/compilers/ghc/8.2.1.nix

  # library instead of the faster but GPLed integer-gmp library.
-, enableIntegerSimple ? false, gmp
+  enableIntegerSimple ? targetPlatform != hostPlatform


Why? Is it hard to build cross-gmp?

Honestly, I probably skipped it at first issue before, and never returned. Good catch.

shlevy · 2017-06-24T00:44:55Z

pkgs/development/compilers/ghc/8.2.1.nix

+, # If enabled, use -fPIC when compiling static libs.
+  enableRelocatedStaticLibs ? targetPlatform != hostPlatform
+
+, # TODO: Make false by default


Why is this a todo instead of just... doing it? 😁

Not sure what this is anymore, but there's no more TODO 😁

shlevy · 2017-06-24T00:45:27Z

pkgs/development/compilers/ghc/8.2.1.nix

+, # Whether to build dynamic libs for the standard library (on the target
+  # platform). Static libs are always built.
+  dynamic ? let triple = targetPlatform.config;
+    # On iOS, dynamic linking is not supported


Not supported by ghc or by the target? Because iOS does allow dynamic linking these days.

I forget 😲!

shlevy · 2017-06-24T00:46:41Z

pkgs/development/compilers/ghc/8.2.1.nix

+    "--with-iconv-libraries=${__targetPackages.libiconv}/lib"
+  ] ++ stdenv.lib.optionals (targetPlatform != hostPlatform) [
+
+    # TODO: next rebuild make these unconditional


Just open this against staging and do it now IMO

Ericson2314 · 2017-06-26T16:11:31Z

@shlevy sorry for the delay in responding. If we really want to clean things up to send to staging, it might make sense to do on top of #26805 then. That will allow deduplicating the dependencies too.

shlevy · 2017-06-26T19:06:08Z

@Ericson2314 Your call if you want to delay this until that is in, no rush on my end

0xABAB · 2017-07-02T12:47:35Z

pkgs/development/compilers/ghc/8.2.1.nix

-  preConfigure = commonPreConfigure;
+  #v p dyn
+  preConfigure = stdenv.lib.optionalString (targetPlatform != hostPlatform)''
+    sed 's|#BuildFlavour  = quick-cross|BuildFlavour  = perf-cross|' mk/build.mk.sample > mk/build.mk


This whole section of writing to mk/build.mk can be done with just one I/O operation using sed just as a stream editor, and the HEREDOC feature of bash. I know we all have fast I/O devices, but it would make the code shorter too.

0xABAB · 2017-07-02T12:49:26Z

pkgs/development/compilers/ghc/head.nix

-  preConfigure = commonPreConfigure;
+  #v p dyn
+  preConfigure = stdenv.lib.optionalString (targetPlatform != hostPlatform)''
+    sed 's|#BuildFlavour  = quick-cross|BuildFlavour  = perf-cross|' mk/build.mk.sample > mk/build.mk


These changes will affect the final derivation

Rather than just sticking it on the PATH

That way the next commit can apply a similar diff to each GHC.

GHC currently handles this stuff in a quite non-standard way, basically taking prog var `FOO` to mean `FOO_FROM_TARGET`. It's because it (wrongly) thinks from stage 2's perspective.

There's no reason to wait for non-binary native to *build* cross ghc, but we want a nix-built GHC for Setup.hs or things won't work.

Hello World with ghcHEAD. ghc822 to come after some patches. Android will be turned into real Android...later.

Cabal2nix expects a --compiler flag that contains a Cabal Compiler description. We used to use the compiler's derivation name for this, but this breaks when cross-compiling due to the target suffix. Instead we add an explicit haskellCompilerName attribute to Haskell compiler derivations.

Ericson2314 · 2018-01-22T04:34:38Z

@peti just rebased to fix a conflict. Do the changes look good to you? Builds passed decently before; hope to merge imminently!

peti

Looks good to me.

Ericson2314 · 2018-01-22T20:33:04Z

Yay!!!!!

expipiplus1 · 2018-01-22T20:33:42Z

Woo, congratulations! 🎉

…

On Mon, 22 Jan 2018, 20:33 John Ericson, ***@***.***> wrote: Yay!!!!! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#26799 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA0U3J2NaUtikAyMBdNyqcBLoh-GxMe7ks5tNPCDgaJpZM4OD-Qa> .

peti · 2018-01-23T12:02:35Z

/me wonders what the purpose of that -n is, to be honest. I don't mind the trailing newline in a generated file.

Ericson2314 · 2018-01-31T05:32:42Z

Eh, no real reason. I guess I was diffing things maybe I dunno.

Ericson2314 requested review from peti and shlevy June 23, 2017 19:47

Ericson2314 mentioned this pull request Jun 24, 2017

Make cross compilation elegant #26805

Merged

13 tasks

Ericson2314 added 6.topic: cross-compilation Building packages on a different platform than they will be used on 2.status: work-in-progress labels Jun 24, 2017

shlevy previously requested changes Jun 24, 2017

View reviewed changes

0xABAB reviewed Jul 2, 2017

View reviewed changes

domenkozar added the 6.topic: haskell label Jul 13, 2017

Ericson2314 mentioned this pull request Jul 16, 2017

ghc cross obsidiansystems/nixpkgs#4

Closed

Ericson2314 modified the milestone: 17.09 Aug 16, 2017

Ericson2314 changed the base branch from master to release-17.09 September 8, 2017 16:47

Ericson2314 force-pushed the cross-haskell branch from 59249b9 to 23eea08 Compare September 9, 2017 15:56

Ericson2314 requested a review from FRidh as a code owner September 9, 2017 15:56

Ericson2314 force-pushed the cross-haskell branch 10 times, most recently from 8dcbb41 to fff31cb Compare September 14, 2017 23:17

Ericson2314 requested a review from copumpkin as a code owner September 14, 2017 23:17

Ericson2314 force-pushed the cross-haskell branch 3 times, most recently from 9a9fbb4 to 33b7086 Compare January 19, 2018 21:24

Ericson2314 mentioned this pull request Jan 21, 2018

GHC as a test case for complications with cross-compiling #21874

Closed

Ericson2314 and others added 15 commits January 21, 2018 23:27

ghc: Normalize derivations a bit before cross

5a6b6b4

These changes will affect the final derivation

ghc: echo -n mk/build.mk to avoid extra line

0eb3aca

generic-builder: Make GHC a proper dependency

54ead73

Rather than just sticking it on the PATH

ghc: Remove old cross work

bc16cfc

That way the next commit can apply a similar diff to each GHC.

ghc: Fix cross compilation to work with new system

398ac54

ghc: Predicate libiconv on not using glibc instead of Darwin

e760de8

ghc: Fix env vars and configure flags to be weird

bd0de2f

GHC currently handles this stuff in a quite non-standard way, basically taking prog var `FOO` to mean `FOO_FROM_TARGET`. It's because it (wrongly) thinks from stage 2's perspective.

ghc 8.2.2 & head: Work around ARM bugs in BFD ld by using gold

af03b77

ghc 8.0.2: Patch Cabal so --extra-*-dirs works right

60f45a2

ghc 8.2.2: Add bgamari's STRIP detection fix

23c29eb

ghcHEAD: Bump version ghc-8.5.20180118

a2516ef

ghc 8.4.1: Reformat to move rev into src

8560c02

haskell infra: nativeGhc != ghc.bootPkgs.ghc

8155312

There's no reason to wait for non-binary native to *build* cross ghc, but we want a nix-built GHC for Setup.hs or things won't work.

release-cross: Add a test for Haskell on Raspberry Pi and "Android"

b612597

Hello World with ghcHEAD. ghc822 to come after some patches. Android will be turned into real Android...later.

Ericson2314 force-pushed the cross-haskell branch from 33b7086 to bf68790 Compare January 22, 2018 04:33

peti approved these changes Jan 22, 2018

View reviewed changes

release-cross: Try building on Windows too

13739e6

Ericson2314 merged commit ec0c480 into NixOS:master Jan 22, 2018

Ericson2314 deleted the cross-haskell branch January 22, 2018 20:31

nmattia mentioned this pull request Feb 13, 2018

support buildHaskellPackages attribute typeable/stackage2nix#52

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

haskell infra: Fix cross compilation to work with new system #26799

haskell infra: Fix cross compilation to work with new system #26799

Ericson2314 commented Jun 23, 2017 •

edited

Loading

mention-bot commented Jun 23, 2017

shlevy left a comment

shlevy Jun 24, 2017

Ericson2314 Jan 10, 2018

shlevy Jun 24, 2017

Ericson2314 Jun 26, 2017 •

edited

Loading

Ericson2314 Jan 10, 2018

shlevy Jun 24, 2017

Ericson2314 Jun 26, 2017

Ericson2314 Jan 10, 2018

shlevy Jun 24, 2017

Ericson2314 Jan 10, 2018

shlevy Jun 24, 2017

Ericson2314 Jun 26, 2017

Ericson2314 Jan 10, 2018

shlevy Jun 24, 2017

Ericson2314 Jan 10, 2018

Ericson2314 commented Jun 26, 2017

shlevy commented Jun 26, 2017

0xABAB Jul 2, 2017

0xABAB Jul 2, 2017

Ericson2314 commented Jan 22, 2018

peti left a comment

Ericson2314 commented Jan 22, 2018

expipiplus1 commented Jan 22, 2018 via email

peti commented Jan 23, 2018

Ericson2314 commented Jan 31, 2018

haskell infra: Fix cross compilation to work with new system #26799

haskell infra: Fix cross compilation to work with new system #26799

Conversation

Ericson2314 commented Jun 23, 2017 • edited Loading

Motivation for this change

Things to do

Optional changes

Misc simple things

mention-bot commented Jun 23, 2017

shlevy left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ericson2314 Jun 26, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ericson2314 commented Jun 26, 2017

shlevy commented Jun 26, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ericson2314 commented Jan 22, 2018

peti left a comment

Choose a reason for hiding this comment

Ericson2314 commented Jan 22, 2018

expipiplus1 commented Jan 22, 2018 via email

peti commented Jan 23, 2018

Ericson2314 commented Jan 31, 2018

Ericson2314 commented Jun 23, 2017 •

edited

Loading

Ericson2314 Jun 26, 2017 •

edited

Loading