[RFC 0046] Platform Support Tiers

[7c6f434c]

I hope to have some meta-discussion here so that #31 gets a language to discuss its specific merits.

render link

[Ericson2314]

Could we have an "embedded" level for and https://github.com/NixOS/nixpkgs/blob/master/lib/systems/examples.nix#L102-L113 NixOS/nixpkgs#59285, this is where very few if any packages are expected to work, but we do provide toolchains for users' convenience. This, as the lowest level support by far, should be an auto-accept for just about anything the compiler supports.

[7c6f434c]

Indeed.

[shlevy]

What does this mean?

[7c6f434c]

There are some things (OpenGL, for example) where level of support for Nix-on-non-NixOS-Linux is lower than on NixOS.

[Ericson2314]

Thanks for adding embedded!

[matthewbauer]

My opinion is we need to maintain an official list of what works and what doesn't. We can change this as time goes on, but we should codify and document what we support. For what exists now, my list of tiers would be:

Tier 1

All packages are expected to work.

Available in binary cache + tier 2 + tier 3

• x86_64-linux
• x86_64-darwin
• aarch64-linux

Tier 2

Most packages are expected to work.

Bootstrap tools are available + tier 3.

• armv{6,7,8}*-linux
• mipsel-linux
• i686-linux
• powerpc64le-linux

Tier 3

Some packages are expected to work.

Cross compilation support is available. Toolchains are built in cross-trunk via release-cross.nix.

• aarch64-none
• avr
• arm-none
• i686-mingw32
• i686-none
• x86_64-none
• x86_64-mingw32
• wasm-wasi
• powerpc-none
• powerpcle-none
• riscv32-linux
• riscv64-linux
• sparc-linux
• sparc64-linux
• alpha-linux
• a few more

Tier 4

No packages are expected to work.

Broken or no support is provided, but may be possible.

• aarch64-darwin
• i686-darwin

ABIs

ABIs are a different concern, but still worth addressing. They should map to one tier.

• gnu{,eabi,eabihf} - tier 1
• musl{,eabi,eabihf} - tier 2
• android{,eabi,eabihf} - tier 3
• mingw32 - tier 3
• apple-ios - tier 4

Toolchains

• GCC+Binutils - tier 1
• Clang+ld64 - tier 1
• Clang+binutils - tier 2
• Clang+LLVM - tier 3
• GCC+LLVM - tier 4

[7c6f434c]

@matthewbauer Thanks for a much better enumeration of platforms! I will try to incorporate most of it into the RFC draft. Do you want to be a coauthor of the RFC? Do you want branch access?

I would say that an honest description of what exists now would have x86_64-darwin and aarch64-linux as tier 1.5 — we aim to have everything working, but significantly more breakages happen there than on x86_64-linux.

I wonder what is our static build support? It should be considered an ABI variant with just gnu-static and musl-static as tier 3 ABIs?

[vcunat]

I suspect that "all packages [with matching meta.platforms]" might be a little too hard to fulfill, though it depends on how hard we take the "expected to work" part and/or whether we tweak the exact meaning of meta.platforms.

In any case I expect it will be better to start with this simpler style – and only eventually consider upgrading the approach, say, to (conceptual) mapping: (package, platform) -> (level of support, maintainers). I think it's quite common that (particular) maintainers only really care about some platforms, even though the package often kind-of works on more of them. In general there can be tension when one platform "slows down" another one in some way. There's a bit more discussion also in the thread linked by GitHub just above my post.

[matthewbauer]

@matthewbauer Thanks for a much better enumeration of platforms! I will try to incorporate most of it into the RFC draft. Do you want to be a coauthor of the RFC? Do you want branch access?

Yeah I can coauthor!

I would say that an honest description of what exists now would have x86_64-darwin and aarch64-linux as tier 1.5 — we aim to have everything working, but significantly more breakages happen there than on x86_64-linux.

Yeah I agree that x86_64-linux is the gold standard. But there's no reason we can't support aarch64-linux or x86_64-darwin equally. They both are built fully by Hydra.

I wonder what is our static build support? It should be considered an ABI variant with just gnu-static and musl-static as tier 3 ABIs?

I would consider the static builds to be an overlay - not a different platform. Support is definitely low since we don't have any Hydra jobs for it. Tier 3 for Musl probably makes the most since. My understanding is support is much better for static linking in Musl than Glibc.

[FRidh]

But there's no reason we can't support aarch64-linux or x86_64-darwin equally. They both are built fully by Hydra.

Not every developer has access to those platforms nor interest to support them.

(package, platform) -> (level of support, maintainers) might indeed be too fine-grained for now, however I think we should already consider something like https://nixos.wiki/wiki/User:Profpatsch#nixpkgs_support_matrix
(Was referred to in NixOS/nixpkgs#60005)

[vcunat]

For aarch64-linux there's the community machine and @GrahamcOfBorg, so not too hard if one is interested; macOS is probably hard for those who don't have it. In any case, the worst thing is human time/effort.

[7c6f434c]

@FRidh Then I look at the list of questions and list of platform dimensions in this RFC and find the linked support matrix a bit too much of a simplification…

@matthewbauer I do want to describe the current/near future state — I do not see Darwin rising from tier-1.5 to tier-1 in the next 3 months.

Overlay or whatever, but static builds are a type of compilation target in my opinion.

I tried to merge the list of build types, and I moved things down to reflect the current state. I hope I am not too wrong.

Gave you push access (apparently I can only do it for the entire fork, but whatever), included as a coauthor in the next revision.

[nixos-discourse]

This pull request has been mentioned on Nix community. There might be relevant details there:

https://discourse.nixos.org/t/rfc-45-and-rfc-46-are-open-for-nominations/2821/1

[CrystalGamma]

Would it make sense to track separate "support levels" (which describe how much is expected to work) and "acceptance levels" (which describe how much platform-specific code – be it configuration conditionals, patches or entire packages – will be accepted).

The levels should then be defined such that, given enough development activity, a "support level" corresponding to the "acceptance level" can reasonably be reached with the amount of platform-specific changes permitted by the "acceptance level".

That would allow us to differentiate between platforms that are mature (support ≥ acceptance) or in the process of being developed (support < acceptance) without having x.5 tiers. Though perhaps shorthands like "tier 3+" for "support level 3, acceptance level 2" would still be useful.

Regardless of whether this makes sense, one thing that should be clarified is what actually constitutes a "platform-specific fix": NixOS/nixpkgs#45474 (which originally started this discussion) was about disabling the intel Mesa driver on non-x86, so that Mesa can be built on ppc64le; is that a ppc64le-specific fix? I would argue that it's actually an x86-specific change (even though it served the purpose of fixing the build on other platforms), considering that the intel driver does not work (or at least didn't at the time of that PR, haven't checked since) on other architectures.

[7c6f434c]

@CrystalGamma Well, the only thing I am reasonably confident, is that we should consider at least the questions listed in the general introduction. But there are too many questions to track separately… In some sense, there are three key dimensions: support, acceptance, and how many packages are used are known to work. (Even x86_64-linux is not perfect in that last metric).

I guess if we think in terms of per-direction tiers, acceptance is weaker than both support and coverage, and support can either come as recognition of coverage or because of coordinated effort of enthusiasts donating build cycles etc.

I tried to word that carefully: I think that disabling intel driver on non-x86 is a cleanup of incorrect assumptions that only ARM and x86 exist, and a problem on low-acceptance platform is enough to replace the wrong assumption with a better one.

Maybe a good test is «do we expect the exact same fix to help on currently unsupported platforms?» (be it RISC-V instead of POWER or BSD libc instead of musl). Should we add this into RFC?

[Ericson2314]

do we expect the exact same fix to help on currently unsupported platforms

That's a good question, because the answer is usually "yes"!

[7c6f434c]

@Ericson2314 there is a bit of selection bias — the fixes applied to non-compiler-toolchain packages to fix minority platforms might have to be generic cleanups to stay in the mainline…

[7c6f434c]

I guess another interesting question is whether we want to define a place for the support table to live, if we want the low-acceptance additions to be cheap and only increases in acceptance to require RFCs…

[matthewbauer]

I guess another interesting question is whether we want to define a place for the support table to live, if we want the low-acceptance additions to be cheap and only increases in acceptance to require RFCs…

I think it's definitely important to document what's built on Hydra + what we have bootstrap tools for.

[7c6f434c]

I agree, and maybe this RFC should include the initial list, but where do we want to maintain the updates to the list?

[matthewbauer]

Perhaps here: https://nixos.org/nixpkgs/

Although that's not so visible...

[Mic92]

I don't think so. I don't think anyone would mine if you start a draft pr for the manual.

[7c6f434c]

> * The definition of support tiers and a list of platforms with the corresponding support tiers is added as an appendix to the Nixpkgs manual. The list of platforms is further maintained as a part of the Nixpkgs manual. The platform list in the manual will be initially based on the list in the appendix of the present RFC. Did this already happen? If not is there a way to help with it?

No, not yet, sorry. This requires a bit of writing (to edit the parts of RFC into something that will look natural in the manual) and I now have a sequence of completely unrelated deadlines including writing, and also it looks like the manual is getting some structural/format changes right now which doesn't help given lack of energy to write. Also checking the recent additions to platform list in <nixpkgs>/lib/systems/platforms.nix and <nixpkgs>/lib/systems/doubles.nix and evaluation which of the low tiers should be assigned to them would be needed. If you are willing to write the glue text (probably by composing it together out of pieces of RFC and editing), edit/update the list itself, and PR against Nixpkgs manual (I guess Markdown chapter would be easier because of preservation of some RFC formatting? And after the conversion we will see what happens), please mention me, I will review and (unless there are objections) merge it after some time (probably a bit more than a week).

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/21-05-retrospective/13426/3

[L-as]

Is there a list of the platforms and their tiers other than this RFC?

[rjzak]

@CrystalGamma how's the powerpc64le-linux support looking currently? I can't get things to build on my Power9 system.

[sternenseemann]

how's the powerpc64le-linux support looking currently? I can't get things to build on my Power9 system.

There has been work on this platform recently by mainly @amjoseph-nixpkgs which is available in the unstable nixpkgs/nixos channels. There is no formal support / support requirement at the moment. You can report any specific issues you are encountering over at the nixpkgs issue tracker.

[ghost]

how's the powerpc64le-linux support looking currently? I can't get things to build on my Power9 system.

There has been work on this platform recently by mainly @amjoseph-nixpkgs

By cherry-picking the PRs below (and their dependent PRs) you can cross-compile a NixOS install image for powerpc64le-linux or mips64el-linux from x86-linux:

• nixos/modules/installer/sd-card/sd-image-mips64el.nix: init nixpkgs#194153
• enable NixOS installer for powerpc64le nixpkgs#192672

The powerpc64le-linux image even boots. I haven't tried the mip64el-linux image yet.

In the long term, it would be nice if we could add an (absolutely not channel-blocking in any way, shape or form) Hydra job for these install images. That way people can try NixOS on these machines without having to trust binaries from anywhere other than Hydra.

That's the limit of my ambitions for these platforms. I don't think it makes sense to build anything more than the installer on Hydra. It would be nice if we had a name for this level of functionality; something like "enabled but unsupported".

[rjzak]

I haven't cross-compiling from x86_64 since I have a ppc64le machine I use as my desktop, and Void Linux PPC has a package for Nix, which seems to work (except for getting things to compile due to the stack checking method).

[ghost]

You can native-compile the installer too.

[asymmetric]

Is there a reason why the @NixOS/aarch64-maintainers team was never created?

[zimbatm]

Probably because it wasn't clear who should do it. Let me know who should be on the team and I will create it.

[asymmetric]

Any takers?

[Atemu]

Well, at least we know why it wasn't created now.

[kip93]

Probably because it wasn't clear who should do it. Let me know who should be on the team and I will create it.

Hello!

Heard about this on matrix. Don't have much experience working on nixpkgs itself but I have been using nix for many years now, mostly for aarch64 systems and doing a lot of cross compiling, so maybe I can be of some help here? If so, I'd be more than happy to join and help.

[JulienMalka]

Any takers?

I have a aarch64 macbook running NixOS and just ordered an ampere dev kit, I would be interested if this team was created to be part of it :)

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-08-03-documentation-team-meeting-notes-69/31263/1

[delroth]

Probably because it wasn't clear who should do it. Let me know who should be on the team and I will create it.

I use NixOS on 3 different AArch64 servers, happy to help with aarch64-specific issues.

EDIT: cross-posted your request on #nixos-on-arm:nixos.org since I suspect your lack of volunteers is mostly because nobody is subscribed to this PR thread.

[aforemny]

Any takers?

I am using NixOS on aarch64 machines (laptop, chromebook) and would be interested in becoming a platform maintainer. Could you please add me to the team?