New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fully static Haskell executables - overview issue #43795

Open
nh2 opened this Issue Jul 19, 2018 · 72 comments

Comments

Projects
None yet
@nh2
Copy link
Contributor

nh2 commented Jul 19, 2018

If you just want to build static Haskell executables right now, follow these instructions.

This issue collects/links all issues and ongoing work to make fully-static building of Haskell executables an excellently-working feature in nixpkgs.

I will update here regularly the progress we are making, and sub-goals we encounter on the way that need to be fulfilled to tick off the parent goal.

Obviously contributions to this list are welcome.

Why is static linking desirable?

Static linking means your executable depends on as few things running on the target system as possible.

For Linux, it means that an executable you build will work on any Linux distribution, with essentially infinite forwards-compatibility (because Linux in general does not change its system call interface).

Statically linked executables do not depend on the target system's libc.

Statically linked executable are incredibly easy to deploy (many Go executables are statically linked, which gave them the reputation of being easy to deploy in the devops world).

Statically linked executables can start faster than dynamically linked ones where dynamic linking has to be performed at startup (of course the startup time depends on the number of libraries to link and the amount of symbols contained within).

Statically linked executables are smaller (up to 4x in our mesurements), mostly because GHC's -split-sections has way more effects with them (for unknown reason for now).

Who is working on or contributing to this

Feel free to add yourself!

@nh2

This comment has been minimized.

Copy link
Contributor Author

nh2 commented Jul 19, 2018

Related:

@nh2

This comment has been minimized.

Copy link
Contributor Author

nh2 commented Jul 19, 2018

CC @fosskers from here

@nh2

This comment has been minimized.

Copy link
Contributor Author

nh2 commented Jul 19, 2018

CC @vaibhavsagar whose blog post got me into working on this

@nh2 nh2 referenced this issue Jul 19, 2018

Closed

Feature/no more haskell libs #43713

0 of 9 tasks complete
@puffnfresh

This comment has been minimized.

Copy link
Contributor

puffnfresh commented Jul 19, 2018

I know NixOS is careful about licenses and distribution. I'm thinking about libgmp, which is LGPL:

https://www.gnu.org/licenses/gpl-faq.en.html#LGPLStaticVsDynamic

(1) If you statically link against an LGPL'd library, you must also provide your application in an object (not necessarily source) format, so that a user has the opportunity to modify the library and relink the application.

(2) If you dynamically link against an LGPL'd library already present on the user's computer, you need not convey the library's source. On the other hand, if you yourself convey the executable LGPL'd library along with your application, whether linked with statically or dynamically, you must also convey the library's sources, in one of the ways for which the LGPL provides.

Can we ensure either one of these if we enable caching of static Haskell executables?

@Gabriel439

This comment has been minimized.

Copy link
Contributor

Gabriel439 commented Jul 19, 2018

@puffnfresh: Or you could use a version of GHC with simple-integer to avoid the GMP dependency

@puffnfresh

This comment has been minimized.

Copy link
Contributor

puffnfresh commented Jul 19, 2018

@Gabriel439 yeah, is that what we'll do for redistributing static Haskell executables?

@vaibhavsagar

This comment has been minimized.

Copy link
Contributor

vaibhavsagar commented Jul 19, 2018

This is great, thanks so much for working on this and creating this issue @nh2!

@dezgeg

This comment has been minimized.

Copy link
Contributor

dezgeg commented Jul 19, 2018

Before doing more of this #43524 -style fixing of individual package builds, could someone give a try of https://github.com/endrazine/wcc (which has been packaged in #43014) in converting shared objects to static objects?

@nh2

This comment has been minimized.

Copy link
Contributor Author

nh2 commented Jul 19, 2018

@puffnfresh Every Haskell package already has a license field (upstream and it's available as a field in nixpkgs).

The best solution seems to be to traverse the licenses of all things linked and ensure that Hydra does not build statically linked exes where one of the dependencies is LGPL or stronger (unless the final project itself is also LGPL or stronger anyway).

For executables blocked that way, a fallback should happen where it's checked if the issue goes away if integer-simple was used instead; then Hydra can build it that way.

@nh2

This comment has been minimized.

Copy link
Contributor Author

nh2 commented Jul 19, 2018

Before doing more of this #43524 -style fixing of individual package builds, could someone give a try of https://github.com/endrazine/wcc (which has been packaged in #43014) in converting shared objects to static objects?

@dezgeg It sounds interesting and may be worth a try but there may be the independent questions whether I want "binary black magic" (as wcc calls it) in my production binaries.

Nix is already pretty custom in its build process, I wouldn't want an upstream library author reject my bug report about a segfault with "well you use totally custom tools, if you just used our normally built static libs we would be more eager to look into that". It's a good feature if nix's builds still essentially are a series of "normal" build steps (compiler and linker invocations).

I haven't looked into wcc in detail so I cannot judge its safety, but my intuition tells me I'd rather sponsor Hydra some build servers and HDs than turn dynamic libs into static ones and have the risk of unexpected behaviour.

@jgm

This comment has been minimized.

Copy link

jgm commented Jul 19, 2018

We already provide fully static builds of pandoc with each release, using docker and alpine.
The build process can be found in the linux/ directory in pandoc's repository.

@fosskers

This comment has been minimized.

Copy link

fosskers commented Jul 19, 2018

Thank you for CCing me into this. Could Aura be added to the list of projects above? I haven't yet been able to decipher the instructions for testing it myself, apologies.

@domenkozar

This comment has been minimized.

Copy link
Member

domenkozar commented Jul 19, 2018

I just want to thank you @nh2 this is excellent work :)

@nh2

This comment has been minimized.

Copy link
Contributor Author

nh2 commented Jul 19, 2018

@fosskers Does aura have a nix derivation in hackage-packages that I could use as a base?

Currently most executables I'm trying are already in nixpkgs and then I just override them with statify function to build statically.

I haven't yet been able to decipher the instructions for testing it myself, apologies.

Conceptually very simple, for example for dhall I just added 1 line here:

https://github.com/nh2/static-haskell-nix/blob/ef283274ce193f713082591dd462f4bd3fb4dd1f/survey/default.nix#L98

and then built with

NIX_PATH=nixpkgs=https://github.com/nh2/nixpkgs/archive/925aac04f4ca58aceb83beef18cb7dae0715421b.tar.gz nix-build --no-link survey/default.nix -A working

For some packages it works immediately, for others I have to make some small overrides of their dependencies (see a bit further up in the file), usually to give static libraries for system dependencies.

@Infinisil

This comment has been minimized.

Copy link
Contributor

Infinisil commented Jul 19, 2018

Having discussed it on IRC, here's a way of building all Haskell executables:
https://gist.github.com/Infinisil/3bdb01689b5f84b71f8538f467159692

Just nix-build that file. This includes broken packages by default, because if you want to see which new packages your change breaks, you need to know which ones were broken already (so you should build this once before your change and once after, then compare).

@nh2

This comment has been minimized.

Copy link
Contributor Author

nh2 commented Jul 20, 2018

Having discussed it on IRC, here's a way of building all Haskell executables

I have incorporated (a large part of) that into https://github.com/nh2/static-haskell-nix/blob/09d0eaa605111ea516dfaa0e7341a71ff1a63042/survey/default.nix#L47-L124 now.

So now we can build survey/default.nix -A allStackageExecutables and see (with --keep-going) how many of those build.

If some binary doesn't build because of dependent libraries making problems, those libraries are supposed to be patched here.

Contributors are welcome to help make as many of those build as possible.

@fosskers

This comment has been minimized.

Copy link

fosskers commented Jul 20, 2018

@nh2 Aura isn't yet connected to Nix infrastructure in any way. I suppose I'll pull up my old notes about getting a project set up with Nix and try to build Aura that way at first.

@dezgeg

This comment has been minimized.

Copy link
Contributor

dezgeg commented Jul 20, 2018

It sounds interesting and may be worth a try but there may be the independent questions whether I want "binary black magic" (as wcc calls it) in my production binaries.

Nix is already pretty custom in its build process, I wouldn't want an upstream library author reject my bug report about a segfault with "well you use totally custom tools, if you just used our normally built static libs we would be more eager to look into that". It's a good feature if nix's builds still essentially are a series of "normal" build steps (compiler and linker invocations).

I understand the concern, yes.

my intuition tells me I'd rather sponsor Hydra some build servers and HDs than turn dynamic libs into static ones and have the risk of unexpected behaviour.

The problem isn't the disk space or resource usage but rather the effort of fixing packages one-by-one to support static linking and the cost of maintaining that. But maybe Haskell packages don't use too many native dependencies.

@nh2

This comment has been minimized.

Copy link
Contributor Author

nh2 commented Jul 20, 2018

I have given a first run at building all executables on Stackage, statically.

https://github.com/nh2/static-haskell-nix/blob/09d0eaa605111ea516dfaa0e7341a71ff1a63042/survey/default.nix#L257-L259

See this post for full build outputs.

It took around 3 hours to get there (I built with 5 machines).

The final status line, [2/961/963 built (49 failed), 4245 copied (19215.1 MiB), 210.7 MiB DL] already gives us some information on the success.

The program didn't terminate; right now it's

  • stuck on building darcs-2.14.1
  • dist/build/test-courier/test-courier from the courier package is stuck on a 100% CPU loop for the last 11 hours

Insights:

  • 6 of the 49 failed executables failed because of #43849
    • this can be fixed by either putting Cabal = Cabal_patched; into the haskellPackagesWithLibsReadyForStaticLinking overrides (I would like to avoid it that unpatched Cabal can be used for most packages)
    • or putting useFixedCabal on all the dependencies used in Setup.hs of those 6 packages (as described in the linked ticket, it can be very cumbersome to figure out which are those dependencies)
  • 15 failed because of cannot find -l*, so static libs are missing for those libraries. These are:
    • The libraries in question are:
      • bz2
      • crypto
      • curl
      • expat
      • girepository-1.0
      • glib-2.0
      • gobject-2.0
      • mpfr
      • nettle
      • pcre
      • pq
      • ssl
      • xml2
    • That's not a lot, fixes for this should be done in this section similar to the other libs we already have overridden there.
@nh2

This comment has been minimized.

Copy link
Contributor Author

nh2 commented Jul 20, 2018

But maybe Haskell packages don't use too many native dependencies.

@dezgeg Yes, I think that is accurate.

From what I posted a just above, it looks like most of Stackage's executables will be buildable as long as a set of 15 native libs (13 above and sqlite and lzma which I already have done) are overridden with static support.

@matthewbauer

This comment has been minimized.

Copy link
Member

matthewbauer commented Jul 20, 2018

Great work!

My rule of thumb for static vs shared is:

  • If you're targeting something outside of a Nix store - build static
  • If you're targeting inside of a Nix store - build shared

We pretty much need to support both in Nixpkgs. I can see some of the benefits of always building statically but I think the advantages to shared linking is much greater.

@nh2

This comment has been minimized.

Copy link
Contributor Author

nh2 commented Jul 20, 2018

I've found and PRd a fix for another cabal issue that needs to be merged to make static linking reasonable:

haskell/cabal#5451

This passes --ld-option through to GHC so that we can specify extra options in configureFlags that are needed only for static linking.

I've also added it to the overview in the issue description.

@nh2

This comment has been minimized.

Copy link
Contributor Author

nh2 commented Jul 20, 2018

My rule of thumb for static vs shared is:

If you're targeting something outside of a Nix store - build static
If you're targeting inside of a Nix store - build shared

This makes sense.

We pretty much need to support both in Nixpkgs

Yes. One of my goals is that nixpkgs becomes the building environment which makes it really easy to build any program so that it works on any Linux distribution, forever.

Other Linux distributions make it really hard to build things statically.

I can see some of the benefits of always building statically but I think the advantages to shared linking is much greater.

I guess I agree in general but there are some exceptions / other points, like

  • Haskell libraries are already statically linked today in nixpkgs and because
    • this makes for much better dead-code elimination (this may also apply to other languages)
    • I've seen dynamically-linked Haskell programs with 100s of .so dependencies take up to 2 seconds to show their --help text
  • Nix(OS) users don't really benefit from the "dependencies can be upgraded cheaply" idea (e.g. patching a libc vulnerability takes only a tiny libc update on other distros) because with nix all downstream dependencies are rebuilt and re-shipped anyway even for the smallest change.
  • So they benefit only from the "dependencies can be shared" idea of dynamic linking.

I'd find it very cool if somebody could build a typical NixOS with only static exes and compare what the size difference (and perhaps resident memory difference) is.

@nh2 nh2 referenced this issue Jul 21, 2018

Merged

Optional static libraries for krb5 and openssl #43870

2 of 9 tasks complete

Gabriel439 added a commit to dhall-lang/dhall-haskell that referenced this issue Jul 21, 2018

Build a completely static Dhall executable in CI
This adds a new `dhall-static` target that builds a fully static `dhall`
executable that can be run on any Linux machine (i.e. it is a relocatable
executable that is completely dependency free).  That in turns implies
that even though it is built with Nix it doesn't require that the user
installs Nix to run it (i.e. no dependency on the `/nix/store` or a
Nix installer).  Just copy the standalone executable to any Linux  machine
and it's good to go.

This based on the following work of @nh2:

* NixOS/nixpkgs#43795
* dhall-lang/dhall-lang#192 (comment)

This also bumps the version of `nixpkgs` used for the normal (non-static)
Dhall build to be the closest revision on `nixpkgs` `master` as the one
used by @nh2 in his work.  Once that work is merged into `nixpkgs` `master`
then both builds can use the same revision from `nixpkgs` `master`.
@Ericson2314

This comment has been minimized.

Copy link
Member

Ericson2314 commented Aug 23, 2018

A static-only Nixpkgs need not support as much as regular nixpkgs! meta.platforms trickery can be used to cordon off what's in scope and what's not in scope. It's similar to a cross package set in that regard.

@Gabriel439

This comment has been minimized.

Copy link
Contributor

Gabriel439 commented Aug 23, 2018

Note that the original scope of this issue is just "Fully static Haskell executables" and if the outcome of this work is that most Hackage packages can be built statically that's still a big win in my book

@fosskers

This comment has been minimized.

Copy link

fosskers commented Sep 8, 2018

Aura is finally on Hackage. Given that it'll probably get auto-magically uploaded to Nix, will it automatically benefit from the things discussed in this issue?

At the very least, could it be added to the list of apps that need to be tested (along with stack and pandoc and friends)? The version of aura currently sitting in my ~/.local/bin/ is 28.1mb.

@nh2

This comment has been minimized.

Copy link
Contributor Author

nh2 commented Sep 9, 2018

@fosskers You need to get it on Stackage for it to be imported autmatically with the next Stackage-to-nixpkgs import.

@nh2

This comment has been minimized.

Copy link
Contributor Author

nh2 commented Sep 10, 2018

@fosskers OK, I've managed to build aura statically by importing some of your packages manually and fixing some dependency versions.

See nh2/static-haskell-nix@245cf2b for the changes I had to make. None of that will be needed once it's on Stackage and imported into nixpkgs.

Size:

9.4M	/nix/store/ix7d5fqn39n7ajniws8qhzv2rgjkm6bp-aura-2.0.0/bin/aura

Built binary for download: aura-static-x86_64.zip

@nh2

This comment has been minimized.

Copy link
Contributor Author

nh2 commented Sep 10, 2018

Here also as a proof-of-concept a build with --arg integer-simple true, to show that I can build against integer-simple instead of integer-gmp to avoid the dependency on the LGPL libgmp:
aura-static-no-libgmp-x86_64.zip

This was built against this master commit and shaves off another 300 KB:

9.1M	/nix/store/lkqk56qw2vsvkl97f0n01937dyxai840-aura-2.0.0/bin/aura
@fosskers

This comment has been minimized.

Copy link

fosskers commented Sep 10, 2018

Nice! What a cut-down in size!

...to avoid the dependency on the LGPL ...

Aura is GPL anyway, so perhaps this isn't a big deal?

You need to get it on Stackage...

I'll take care of this tomorrow (Monday) morning.

@nh2

This comment has been minimized.

Copy link
Contributor Author

nh2 commented Sep 10, 2018

Aura is GPL anyway, so perhaps this isn't a big deal?

Oh yeah, this was just to demonstrate that the approach makes it easy to get rid of the gmp dependency for arbitrary executables, in case somebody desires that, addressing #43795 (comment)

@nh2

This comment has been minimized.

Copy link
Contributor Author

nh2 commented Sep 20, 2018

Tangentially, here's an issue with pkgsMusl's musl-gcc still linking in glibc stuff (just for my records): #25178 (comment)

@albertov

This comment has been minimized.

Copy link

albertov commented Oct 25, 2018

I'm trying to build a GHC version for system armv7l-linux with Musl to generate static Haskell executables on that platform (through QEMU) but hitting a strange issue with ld.gold as it refuses to work with Musl (works fine in Alpine though).

This prevents building GHC for armv7l since GHC needs ld.goldon that platform due to https://sourceware.org/bugzilla/show_bug.cgi?id=16177. The problem also occurs in x86_64 which is not a blocker but prevents ld.gold to be used to improve linking time.

I've created a ticket for this issue here but cross-posting it here since it is related and hopefully catches more eye balls: #49071

Has anyone seen this problem before? Any idea what could be causing it or how to debug it? The linker does not emit any useful error message.

@NorfairKing

This comment has been minimized.

Copy link

NorfairKing commented Oct 25, 2018

cc @MasseR

@fosskers

This comment has been minimized.

Copy link

fosskers commented Dec 5, 2018

Any news?

@Gabriel439

This comment has been minimized.

Copy link
Contributor

Gabriel439 commented Dec 5, 2018

@fosskers: I believe this is mainly waiting on these two pull requests against Cabal to be merged:

Everything else appears to be ready

@nh2

This comment has been minimized.

Copy link
Contributor Author

nh2 commented Dec 5, 2018

There's actually been lots of progress:

  • The current official static stack release is built using this stuff.
  • #48803 is an effort to move more of static-haskell-nix directly into nixpkgs. But it continues to be ready now when you need to build something statically.
  • On MuniHac people worked on integer-openssl, which is a BSD-licensed integer-gmp alternative.

That said, merging the Cabal patches would be something extremely beneficial to reduce long-term maintenance overhead.

@nh2

This comment has been minimized.

Copy link
Contributor Author

nh2 commented Dec 6, 2018

@nh2

This comment has been minimized.

Copy link
Contributor Author

nh2 commented Dec 6, 2018

PR #48803 - Adding pkgsStatic: a fully static overlay has been merged 20 hours ago!

@nh2

This comment has been minimized.

Copy link
Contributor Author

nh2 commented Dec 6, 2018

  • The ncurses .a and tinfo fix has been merged to master

is now true so the ncurses part has been crossed off completely.

@fosskers

This comment has been minimized.

Copy link

fosskers commented Dec 6, 2018

Wow awesome guys, thanks for the updates.

@cdepillabout

This comment has been minimized.

Copy link
Contributor

cdepillabout commented Dec 6, 2018

Yeah, thanks for all your work! I've been following this issue with some interest for a while now, and I am looking forward to the day when it can finally be closed!

@nh2

This comment has been minimized.

Copy link
Contributor Author

nh2 commented Feb 7, 2019

Update:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment