Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide middle-ground overlay between pkgsMusl and pkgsStatic #61575

Open
nh2 opened this issue May 16, 2019 · 10 comments

Comments

Projects
None yet
4 participants
@nh2
Copy link
Contributor

commented May 16, 2019

Issue description

For my work on static-haskell-nix I need an overlay that has both .so files and .a files.

This is because I need .as for linking Haskell exes statically, but .so files for executing TemplateHaskell.

Right now pkgsStatic not only adds .a files, but also disables .so files:

Proposal

I think we should have the following stack of overlays:

pkgsMusl <- pkgsStatic <- pkgsStaticOnly
  • pkgsMusl is what it is now: All packages using musl libc
  • pkgsStatic should be the one that has .a files added to all libraries (in addition to .so files), and all binaries (including Haskell ones) linked statically where possible
  • pkgsStaticOnly should be what pkgsStatic is now: Having only static libraries and binaries, getting rid of .so files.

CC @matthewbauer @Ericson2314 @dtzWill @nmattia

@Ericson2314

This comment has been minimized.

Copy link
Member

commented May 16, 2019

So what's the deal with static libs and GHCi/TH? It used to work? CC @angerman

@matthewbauer

This comment has been minimized.

Copy link
Member

commented May 16, 2019

Yeah I was hoping we could use -fexternal-interpreter for this case.

@nh2

This comment has been minimized.

Copy link
Contributor Author

commented May 16, 2019

Related: I asked on #ghc:

Can ghc currently load static libs when evaluating TH, e.g. libgmp.a? My musl based builds fail in pkgsStatic when I don't have libgmp.so

@angerman said:

yes it can. If you disable the DYNAMIC define.
there is a call called loadDLL or something that just defaults to shared objects unless ghc is explicitly set to not use dynamic libraries.
angerman
https://github.com/ghc/ghc/blob/610ec224a49e092c802a336570fd9613ea15ef3c/compiler/ghci/Linker.hs#L1467

@cocreature said:

angerman: that line only applies to Haskell libraries (which funnily enough is why you can get GHC to load static Haskell libraries in TH by putting them in extra-libraries instead of hs-libraries) so it shouldn’t be relevant for libgmp

we actually use the extra-libraries trick at work (for various reason that are probably better explained at ZuriHac :)) and surprisingly it hasn’t exploded so far

the logic for C libraries should fall back to static libs afaik so not sure what issue nh2 is seeing

it looks like if findArchive fails it will fall into the assumeDll so maybe GHC doesn’t find the libgmp.a that nh2 has lying around?

@matthewbauer

This comment has been minimized.

Copy link
Member

commented May 16, 2019

My main concern with building both dynamic and static is that build systems will get messed up when they see both. They might try to use the wrong one and do the wrong thing. It's better they fail than do the wrong thing here. It's very similar to why we don't build multi-target things as well. Nix means "nothing", not everything 😸 .

@nh2

This comment has been minimized.

Copy link
Contributor Author

commented May 16, 2019

build systems will get messed up when they see both

@matthewbauer In what situation does that happen though? Most build systems have a flag that controls whether you want static or dynamic exes. If you link with -static, it's going to use .a files, or fail complaining (or at least that's what I've observed so far).

pkgsStaticOnly would still be as you desire, but having the middle thing as well does allow us to build non-cross static executables that need .sos on the way there quite effortlessly today.

@Ericson2314

This comment has been minimized.

Copy link
Member

commented May 16, 2019

So from the GHC quotes this does sound quite fixable?

@angerman

This comment has been minimized.

Copy link
Contributor

commented May 16, 2019

So from the GHC quotes this does sound quite fixable?

Everything is fixable if you apply enough WD40 and/or duct tape.

@nh2

This comment has been minimized.

Copy link
Contributor Author

commented May 16, 2019

So from the GHC quotes this does sound quite fixable?

@Ericson2314 It is possible, but what I'm looking for is a way to get static Linux builds immediately using only as-standard-as-possible approaches without teaching GHC new things. The easiest way to get there is to have .so + .a. Many projects already benefit from distributing static binaries this way, including dhall, stack and others.

That is not to say there's no merit in also working on the other approaches, but because these things have users already right now, I'd like to move them to nixpkgs rather sooner than later, because maintaining all the C library overrides to add .as in static-haskell-nix is an effort that would go away if we could share those overrides with pkgsStatic.

@Ericson2314

This comment has been minimized.

Copy link
Member

commented May 17, 2019

@nh2 I'm mainly just worried about arbitrary non-Haskell packages misbehaving when both are available. When I say it sounds easy to change GHC, I mean easy to write a patch and backport it to use in nixpkgs immediately. Based only my experience working on Cabal in the past, and working on GHC now, I'd say landing GHC patches is easier in general too, lest your experience with the Cabal patch makes it seem that any upstream change is invariably a chore.

@nh2

This comment has been minimized.

Copy link
Contributor Author

commented May 17, 2019

I'm mainly just worried about arbitrary non-Haskell packages misbehaving when both are available.

@Ericson2314 Do you have an example? What kind of scenario do you have in mind?

When I say it sounds easy to change GHC, I mean easy to write a patch and backport it to use in nixpkgs immediately.

I don't doubt that we can do thinkgs quickly in both GHC and nixpkgs, but I'm unclear on what exactly we need to do on the GHC side to make it never look at .so files, and who can work on that in the immediate future.

In general, is there a drawback to what I propose (having one place / adapter / overlay that simply enables .as for as much as possible)? The concerns you may have about packages misbehaving would not apply to what's now pkgsStatic, as that one would be unmodified (turning .sos off).

@nh2 nh2 referenced this issue May 18, 2019

Open

Fully static Haskell executables - overview issue #43795

32 of 47 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.