Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RFC 0023] Support musl libc #23

Merged
merged 9 commits into from Jun 28, 2018

Conversation

Projects
None yet
@dtzWill
Copy link
Contributor

dtzWill commented Feb 20, 2018

WIP, but comments and discussion welcome throughout the process :).

RFC 23 - support musl libc (rendered)

@dtzWill dtzWill changed the title [WIP] Support musl libc [RFC 0023] Support musl libc Feb 20, 2018

@dtzWill dtzWill referenced this pull request Feb 20, 2018

Merged

musl-cross, native-musl, cross fixes, oh my! #34645

1 of 3 tasks complete
@shlevy

This comment has been minimized.

Copy link
Member

shlevy commented Feb 20, 2018

👍 from me.

@cstrahan

This comment has been minimized.

Copy link

cstrahan commented Feb 20, 2018

This almost perfectly matches my own thoughts. I’m all for it. 👍

@7c6f434c

This comment has been minimized.

Copy link
Member

7c6f434c commented Feb 20, 2018

Given that the only hardware platform that >90% have with root access is x86_64, and that we don't have a free software Darwin image that can be booted in a VM, doing portability fixes on x86_64-linux does sound appealing.

And with the current amount of effort towards cross-compilation, don't we have any cross-targets in use by Nixpkgs users where musl is the preferred libc?

@edolstra

This comment has been minimized.

Copy link
Member

edolstra commented Feb 20, 2018

IMHO it's best to keep musl support in a branch or overlay so that the effort of maintaining musl-specific patches is not inflicted on package maintainers who don't care about musl.

(The same applies to RISC-V support BTW, which really should also be the subject of a RFC.)

@coretemp

This comment has been minimized.

Copy link

coretemp commented Feb 20, 2018

I am in favor of a design where everything is in one repo where a broken musl doesn't impact glibc package users. I am not in favor of a branch based approach or overlay. glibc should only have a temporary special status in this regard, such that at some point we might even switch to musl as the default library.

@zimbatm

This comment has been minimized.

Copy link
Member

zimbatm commented Feb 20, 2018

Overlays work best when the package set is different. In the case of musl it seems like it would be hard to reach into all the packages with overrides. Nix is quite composable but some times it's not possible to pry open a derivation enough to replace the inner parts (like when a function is hiding the scope).

How about accepting the musl patches but not making it a CI failure? It's not official but it's already the case that we have supported platforms and unsupported ones in nixpkgs.

@zimbatm

This comment has been minimized.

Copy link
Member

zimbatm commented Feb 20, 2018

I mean as part of an unsupported platform we would set the expectations: it's okay to break the musl patches as part of the maintenance cycle. The musl team can have their own build and test infrastructure to sweep after breaking changes.

@Ericson2314

This comment has been minimized.

Copy link
Member

Ericson2314 commented Feb 20, 2018

I'm super for it. And I'd like some cross platform to in fact be required by CI too. I think it's good that different platforms make packaging harder, it keeps us and upstream honest. Lastly, Musl, with it's minimal set of features, is a great test because if a package is built with musl, it can probably be built with anything.

Just to reiterate, the key to writing portable software is parametricity: don't add endless #ifdefs or conditionals in Nix, but abstract over platform differences as far upstream as possible so downstream need not care. With this principle, more platforms is not linearly more work.

@7c6f434c

This comment has been minimized.

Copy link
Member

7c6f434c commented Feb 20, 2018

@edolstra are you also against the cross-compilation support in the main repository?

Making a call on macOS failures is already needed, and with musl I can at least reproduce the problem locally and see if it is easy to fix; so it is just the question about the level of expected stability.

@zimbatm

This comment has been minimized.

Copy link
Member

zimbatm commented Feb 20, 2018

@Ericson2314 it's like every platform. First it should go through a trial phase where compilation success is not required. Once it has been demonstrated as a viable target it can be promoted to the supported platforms.

The question is, who is responsible for deciding which platforms are supported or not :)

@Ericson2314

This comment has been minimized.

Copy link
Member

Ericson2314 commented Feb 20, 2018

@zimbatm Agreed, I think a release.nix + release-cross.nix merge should be done by 18.03 with no new required-to-pass derivations, as exactly that second step.

@zimbatm

This comment has been minimized.

Copy link
Member

zimbatm commented Feb 20, 2018

Alright let's do this. Any objections @edolstra ?

@Ericson2314

This comment has been minimized.

Copy link
Member

Ericson2314 commented Feb 20, 2018

I should add that @dtzWill was a bit more to the point in the original RFC than me pontificating on parametricity above. I think prior portability efforts were a bit more "conditional-explosion-y" than the current ones: the patches we're capable of landing today really do improve the software in code quality too, and I'd love to upstream them. Given the necessary ugliness of earlier efforts, I understand and sympathize @edolstra's reservations. Given the elegance we can get away with now, I don't think those concerns still apply.

@Mic92 Mic92 referenced this pull request Feb 20, 2018

Merged

monit: cross-compile, and make openssl optional #35248

2 of 8 tasks complete
@grahamc

This comment has been minimized.

Copy link
Member

grahamc commented Feb 20, 2018

I've noticed the RFC as-is doesn't go much in to how maintenance will work:

  • how we test and verify patches work
  • how well will musl be supported in the end
  • do most packages work out of the box with musl?
  • what impact does this have on Hydra?
  • what impact does this have on ofborg?
  • how do we detect if a patch is breaking musl support?

We should also take care to remember this adds a bit more complexity to many derivations.

Does musl work on macOS? What happens if someone tries to use musl on MacOS in Nixpkgs?

feature: musl-libc
start-date: 2018-02-19
author: Will Dietz
co-authors: (find a buddy later to help our with the RFC)

This comment has been minimized.

@grahamc

grahamc Feb 20, 2018

Member

I'm concerned that a whole new libc only has one "sponsor" here.

This comment has been minimized.

@shlevy

shlevy Feb 20, 2018

Member

I'd happily cosponsor.

This comment has been minimized.

@grahamc

grahamc Feb 20, 2018

Member

The more the merrier, I think! This writes down who can be pinged for future musl problems.

This comment has been minimized.

@AndersonTorres

AndersonTorres Feb 21, 2018

Would it be a hard task?

@Ericson2314

This comment has been minimized.

Copy link
Member

Ericson2314 commented Feb 21, 2018

@graham I think the idea is that initially, nothing is supported, but merely allowed to exist.

how we test and verify patches work

The only way to is to build, but the package maintainer isn't obligated to do this.

how well will musl be supported in the end

Depends how the experiment goes

do most packages work out of the box with musl?

Yes. Packages that don't will probably be eventually broken on glibc as it becomes more standards-enforcing. (I didn't know this until I read the RFC. This is great news as too much of GNU doesn't require opt-in to extra GNU-specific features.)

What impact does this have on Hydra / of borg?

Nothing. Well almost nothing; there are a few odd jobs in release-cross.nix.

how do we detect if a patch is breaking musl support?

We don't need to for an unsupported experiment.

@grahamc

This comment has been minimized.

Copy link
Member

grahamc commented Feb 21, 2018

the idea is that initially, nothing is supported, but merely allowed to exist.
Depends how the experiment goes

I don't love the idea of having unsupported, unexercised code sitting in Nixpkgs. What is the point of having it in nixpkgs, if it is allowed to be broken by people who don't even know it is being broken? At that level of "unsupported", there is very little reason to have it in the repo. Nixpkgs receives almost 500 commits a week, and a lot of rot can set in over that time.

I think if we're going to add major new support, part of the experiment should be at least diagnostics-level checking of the code per PR.

If you disagree with this, please pretend we're a few months down the line and musl is a resounding success and I re-asked all those questions.

P.S. I feel positive about this project.

I believe the linked implementation from current nixpkgs master is perhaps
the best embodiment of "Detailed Design"?

Linky: [bd7d5b365799a145717d31122ebfd24b51fea117](https://github.com/NixOS/nixpkgs/commit/bd7d5b365799a145717d31122ebfd24b51fea117)

This comment has been minimized.

@grahamc

grahamc Feb 21, 2018

Member

I suspect this link is wrong :)

This comment has been minimized.

@dtzWill

dtzWill Feb 21, 2018

Author Contributor

I think it's right, it's just revision of master at time of writing.

This comment has been minimized.

@Ericson2314

Ericson2314 Feb 21, 2018

Member

Maybe a diff or merge commit would be better?

author: Will Dietz
co-authors: (find a buddy later to help our with the RFC)
author: Will Dietz (@dtzWill)
co-authors: Shea Levy (@shlevy)

This comment has been minimized.

@grahamc
@shlevy

This comment has been minimized.

Copy link
Member

shlevy commented Feb 21, 2018

A new libc or port is simply not something that can realistically live out of tree, overlays etc. aren't expressive enough. Saying that we need a full support story before we can start work proving viability, attracting maintainers, and building userbase is effectively saying "go fork nixpkgs, this is never going to happen upstream."

Or if you disagree, how exactly do you propose we go about this kind of thing such that we have a realistic chance of ever getting it merged, given the above constraints?

@shlevy

This comment has been minimized.

Copy link
Member

shlevy commented Feb 21, 2018

The point of having it in nixpkgs is so that people can use it, and iterate on it. People undertaking these efforts understand there is burden on them to do whatever form of CI they need on their own resources until there is enough community support for more. If things bitrot, that's what git rm is for.

@shlevy

This comment has been minimized.

Copy link
Member

shlevy commented Feb 21, 2018

I hesitate to post this because I think it's a worse solution than allowing it upstream, but: if the ultimate answer here is no (and, setting this particular RFC aside, we don't seem to have any way to get to an ultimate answer in these...), then I'll (on my own fork) start nixpkgs-next à la linux-next, where all sorts of experiments are regularly merged on top of master and it's expected that the individual experiment maintainer makes sure things don't break for everyone else (enforced by being removed from the repo list until fixes are in) and users who want to take advantage can use a common place that's kept up-to-date with master.

To reiterate, this is a worse solution than finding a way to let this kind of work happen upstream. We're not the kernel community, and the last thing we need is further impaired discoverability, further divergence in possible user configurations, and more third-party infrastructure. But I'm tired of ideas being shot down because of potential maintenance burden without any viable alternative existing, so there's a viable alternative.

@zimbatm

This comment has been minimized.

Copy link
Member

zimbatm commented Feb 27, 2018

I can see three main uses of musl: embedded (NixWRT?), smaller docker images, publishing static binaries (for non-nix targets). All of these seem like good things to have to increase Nix's popularity.

If there are 3+ people that pledge to create a self-managed musl team, I think that should be enough to accept the RFC.

@shlevy

This comment has been minimized.

Copy link
Member

shlevy commented Feb 27, 2018

@zimbatm Agreed on both counts. I'm willing to be part of that team in whatever capacity @dtzWill thinks makes sense.

@domenkozar

This comment has been minimized.

Copy link
Member

domenkozar commented Feb 27, 2018

You can also count on IOHK helping maintain musl (as a consumer of the work).

About the integration: I'd start with one codebase (nixpkgs master) and decide otherwise if it's too much conditionalization once the prototype is ready for review.

@shlevy

This comment has been minimized.

Copy link
Member

shlevy commented Feb 27, 2018

OK, I think this is a go then. @dtzWill any modifications before we merge?

@Ericson2314

This comment has been minimized.

Copy link
Member

Ericson2314 commented Feb 27, 2018

I'd say double check with @grahamc that his questions are addressed. That should be quick back and forth.

@grahamc

This comment has been minimized.

Copy link
Member

grahamc commented Feb 27, 2018

  1. "musl team" is reponsible, to be formed please form that as part of this RFC. It sounds like you have the team here, it just needs to be written down. We could create a proper GitHub team handle that can be pinged, too.
  2. These changes usually can be safely applied on all platforms (although sometimes they are not for rebuild reasons) I'd be inclined to just do it anyway, especially if these changes are largely fixes improving compliance and correctness resulting in higher-quality programs" holds true. Reducing the confusion and variability and chances some other patch only breaks on musl. Essentially, reducing the cyclomatic complexity of nixpkgs.
  3. Please open an issue on https://github.com/nixos/ofborg with a details description (or even, a PR!) of how to add it to the evaluation checker. Here is probably the best place to look on how the eval checks are done cross-platform: https://github.com/NixOS/ofborg/blob/released/ofborg/src/outpaths.nix

Following these, as far as I'm concerned,

a jet airplane taking off, implying approval

@7c6f434c

This comment has been minimized.

Copy link
Member

7c6f434c commented Feb 27, 2018

I would say that if a musl patch is changing just the assumption «Linux = Android xor glibc», it could be OK to delay its application on Darwin to reduce the number of rebuilds, but eventually applying it could indeed be a good idea.

@Mic92 Mic92 referenced this pull request Feb 28, 2018

Merged

[RFC 0025] Nix Core Team #25

@telent

This comment has been minimized.

Copy link

telent commented Feb 28, 2018

As a non-contributor, I don't know if my opinion carries any weight at all, but here it is anyway:

  • I don't have a strong opinion on process or responsibilities, but if it can be made to work then as NixWRT wrangler I would find it absolutely marvellous: switching my (admittedly small) package set from glibc to musl was accomplished in a single evening and the only change needed in nixpkgs was actually a cross-compilation fix not a musl fix. It opens up so many opportunities for using the Nix package collection in low-resource/embedded scenarios.

  • Despite that, I would not be in favour of this if I thought that it was going to be a gateway to "combinatorial explosion hell" - adding ugly conditional fixes to packages to get them to build with a feature-impoverished or buggy C library. But by all accounts it's not that and could even be looked on as a form of "linter" - improving the compatibility and standards-compliance of the code in the package collection.

tl;dr I can't speak to whether this approach to supporting musl-in-nixpkgs is the right one, but I can say I'd definitely be a user of musl-in-nixpkgs and so the need addressed by this RFC is (for me, anyway) real.

@dtzWill

This comment has been minimized.

Copy link
Contributor Author

dtzWill commented Mar 1, 2018

Alright, I suppose it's ready? 😇

@grahamc I liked the team idea so I put it in the RFC, hope that's alright 👍.

@rasendubi

This comment has been minimized.

Copy link
Member

rasendubi commented Mar 1, 2018

I'm interested in being the part of the musl team, too.

@copumpkin

This comment has been minimized.

Copy link
Member

copumpkin commented Mar 1, 2018

I would not be in favour of this if I thought that it was going to be a gateway to "combinatorial explosion hell" - adding ugly conditional fixes to packages to get them to build with a feature-impoverished or buggy C library. But by all accounts it's not that and could even be looked on as a form of "linter" - improving the compatibility and standards-compliance of the code in the package collection.

Hear, hear!

Merge pull request #1 from rasendubi/patch-1
Add myself to the "musl team"
@dtzWill

This comment has been minimized.

Copy link
Contributor Author

dtzWill commented Mar 1, 2018

(... I just switched my NixOS profile to use musl-built Nix 2 😁 ...)

Note that this currently requires a small patch to Nix to avoid stack-allocating large buffers: dtzWill/nix@b538660

@vcunat

This comment has been minimized.

Copy link
Member

vcunat commented Mar 5, 2018

Just going to say that I'm really hoping that if we'll accept this RFC, we'll do so by 18.03.

@Ericson2314: do you mean that you want the musl-support patches to keep getting cherry-picked to the stable 18.03 branch already? EDIT: 18.03 is very close to (belated) branching ATM.

@Ericson2314

This comment has been minimized.

Copy link
Member

Ericson2314 commented Mar 5, 2018

@vcunat I guess I didn't think that far; correct me if I'm wrong @dtzWill, but I thought stream of musl fixes and slowed to a trickle :).

All I was thinking is if we're going to go through with this, it would be a cool thing to publicize (somewhat, is it is technically an experiment). If we end up going ahead but only decide after after 18.03, we miss a good window to give the experiment more visibility.

@zimbatm

This comment has been minimized.

Copy link
Member

zimbatm commented Mar 6, 2018

Until we have a "nixpkgs team" to make a decision on this, I propose to delegate this decision to the release team: @vcunat and @fpletz

@dtzWill dtzWill referenced this pull request Mar 23, 2018

Open

[WIP] systemd: patch to build on musl #37715

2 of 8 tasks complete

@edolstra edolstra merged commit 13b5084 into NixOS:master Jun 28, 2018

@edolstra

This comment has been minimized.

Copy link
Member

edolstra commented Jun 28, 2018

Merged this because it reflects the current situation.

@nh2

This comment has been minimized.

Copy link

nh2 commented Sep 11, 2018

It seems the 1.1.19 -> 1.1.20 upgrade broke GCC for me: NixOS/nixpkgs#46064 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.