LLVM needs additional maintainers (mainly for updates)

[primeos]

Note: I'll edit this post to keep it up-to-date.

We do have LLVM maintainers that improve the Nix expressions, handle PRs, etc. but we do severely lack maintainers to package new LLVM versions. Adding new LLVM versions is currently handled by me but I neither have enough time nor motivation to handle LLVM (which is also why I often (have to) take some shortcuts, e.g.: https://github.com/NixOS/nixpkgs/pull/162423/files). I'm basically only doing that work because Chromium requires bleeding-edge LLVM versions to compile (it's currently already difficult to compile Chromium with LLVM 14 even tough there isn't even a RC for LLVM 15 out - Google simply chooses to pick Git commits from the main branch...).

The most relevant tasks that need help are:

1. Keeping llvmPackages_git up-to-date (this is mostly important to enable us to quickly package new LLVM versions as soon as the first RC is out without having to put in too much work at once). This requires most of the time. It currently requires resolving a lot of merge conflicts due to our GNU install dirs patches (it wasn't really difficult so far and didn't require knowing the code but it takes a fair bit of time as there are a lot of those patches and often a few other build failures that require fixing).
   • llvmPackages_git is currently severely out-of-date and it would probably be best to replace it with the content of llvmPackages_14 first (rm -r pkgs/development/compilers/llvm/git/ && cp -r pkgs/development/compilers/llvm/14/ pkgs/development/compilers/llvm/git but one needs to check the Git log first)
2. Packaging new LLVM versions as soon as the first RC is out (that's pretty simple/straightforward as it's basically just copying llvmPackages_git and switching to the right version, e.g.: llvmPackages_14: init at 2022-01-07 #162104 (I prefer doing it in two commits but one could also diff it against llvmPackages_git)).
   • LLVM recently switched to a bi-weekly release schedule (vs. just 1-2 additional releases per major version) so this will also require more time/effort. But we might not even want to package every bi-weekly release (especially since our staging workflow is probably to slow for that and it should cause quite a few rebuilds if llvmPackages_latest points to it).
3. Updating llvmPackages_latest a super easy change (just one line) but testing is challenging (a full nixpkgs-review might be doable but one needs to exclude the transient dependencies from Mesa - using Hydra might be easier).
   • This work is already behind: llvmPackages_latest: 13 -> 14 #165701.
4. And of course we could also really need testers/maintainers for other platforms (AArch64, Darwin, etc.).
5. Remove old LLVM versions from Nixpkgs. We currently package everything from version 5 (released in 2017) up to version 14 (i.e., 10 different versions - and that number is likely growing).

LLVM is one of currently five release critical packages and some notable packages that depend on recent versions are:

• Chromium (often even requires RCs - currently it would require an unreleased LLVM version for PGO and some build flags)
• Mesa (uses llvmPackages_latest and is responsible for most of the rebuilds)
• Rust (already depends on llvmPackages_14)

And for documentation:

• We currently have two update scripts
  • pkgs/development/compilers/llvm/update.sh
  • pkgs/development/compilers/llvm/update-git.py

I'll try to answer questions / help during the transition but tbh I don't know that much about LLVM (just enough to update it without too much breakage).

cc existing LLVM maintainers: @lovek323 @7c6f434c @dtzWill @primeos
And (AFAIK) @sternenseemann and @Ericson2314 do most of the Nix expression maintenance (refactorings/improvements/cleanups/cross-fixes/etc.)

[sternenseemann]

Maybe we should try making the experience of bumping llvmPackages_git better as a first step and try to keep it up to date. Ideally, then, adding the latest version is just copying a bunch of files and selecting a tag instead of a commit. An idea I have would be to have some kind of script which bumps llvmPackages_git, tries to build chromium and then creates a PR for it. Having a machine that does this weekly or so would be awesome (I'm unsure if we could get @ryantm's nixpkgs-update to do it, but I think even if we could, the machine would surely die or timeout the nixpkgs-review of the change).

It would be lovely if we could delete older LLVM version or alternatively consolidate the expressions for different versions, but I'm afraid that that's not attainable without a lot of work. We are partially responsible for this as well, since we force divergence in the expressions by upstreaming stuff that makes packaging in nixpkgs easier :)

I'm happy to be a more official LLVM maintainer, but unsure how much actual time I can contribute, as a lot of it already goes into Haskell.

Edit: Also: Maybe we want to make a “formal” llvm team.

[primeos]

Maybe we should try making the experience of bumping llvmPackages_git better as a first step and try to keep it up to date.

Yes, +1. Unfortunately, I'm not sure if that's really possible though. For me the main annoyance was the addition of the GNU install dirs patches. Those regularly cause conflicts and then I have to copy them over to Git, resolve the conflicts, copy the results back, etc. At least for me that required most of the time in the past and is currently keeping me from giving it another go (llvmPackages_git is already so outdated that there are likely a lot of conflicts - the required changes for the last major bump can be seen here: https://github.com/NixOS/nixpkgs/pull/154465/files).

It would be lovely if we could delete older LLVM version

That's also an additional task that I missed. We have quite a lot of old versions... :o

I'm happy to be a more official LLVM maintainer

IMO it still makes perfect sense to add you and @Ericson2314 to the maintainer list (you don't even have to put in more work - you're already maintaining LLVM and it would simply reflect the current state of things).
As I've written before (#154465 (comment)):

AFAIK you're currently doing by far most of the work anyway and usually end up reviewing other PRs as well.

Feel free to submit a PR at any time ;)

Btw @lovek323 @7c6f434c @dtzWill: What's your current status? IIRC I haven't seen your handles on LLVM PRs/issues often anymore but I could be wrong. Are you still interested in maintaining LLVM?

Also: Maybe we want to make a “formal” llvm team.

+1, SGTM.

[7c6f434c]

I was actually doing something with LLVM packaging long ago. Nowadays I just sometimes merge additions that seem very safe and pretty clean but forgotten in the PR flood. (So, no useful opinion on best changes to the current situation, and feel free to remove or not remove me from the list)

[lovek323]

I'm not sure how much time I can devote to this as well – the git conflict resolution seems like it may require deeper knowledge than I currently have. Could you put some steps down that I could try to follow to see if I have the expertise to do it?

[dtzWill]

Thank you very much for starting this discussion. I agree the importance of LLVM and packaging it /well/ and dutifully is not reflected in the attention we (collectively) are giving it. As you say, this is critical, and also important to me.

I'm basically only doing that work because Chromium requires bleeding-edge LLVM versions to compile (it's currently already difficult to compile Chromium with LLVM 14 even tough there isn't even a RC for LLVM 15 out - Google simply chooses to pick Git commits from the main branch...).

Sigh re:chromium being so intensely aggressive chasing latest LLVM. Thank you for your efforts on this front!

1. Keeping llvmPackages_git up-to-date (this is mostly important to enable us to quickly package new LLVM versions as soon as the first RC is out without having to put in too much work at once). This requires most of the time. It currently requires resolving a lot of merge conflicts due to our GNU install dirs patches

The GNU install dirs patches are difficult, I agree. The git stuff isn't so bad (although this last instance was rough) I think they're the biggest pain point re:getting things building (vetting/testing aside for now). Hopefully the rest of what we need can be upstreamed soon, or is that not likely? Can we push on this? Or are there other options/approaches we can take? If that's not in the cards, how about we work on documenting what they do and why they're needed (or what to check to ensure things are as-expected) so that it's easier to port them with confidence?
They seem to have something to do with ensuring our outputs each have what they "should", but that isn't always clear when diffing outputs across versions-- as LLVM changes what it installs where or adds/removes files, or changes its build system code so a better understanding would be useful, IMHO.

• LLVM recently switched to a bi-weekly release schedule (vs. just 1-2 additional releases per major version) so this will also require more time/effort. But we might not even want to package every bi-weekly release (especially since our staging workflow is probably to slow for that and it should cause quite a few rebuilds if llvmPackages_latest points to it).

• Updating llvmPackages_latest a super easy change (just one line) but testing is challenging (a full nixpkgs-review might be doable but one needs to exclude the transient dependencies from Mesa - using Hydra might be easier).

  • This work is already behind:

These can (and ideally they would--if we moved to latest as soon as was safe) be related issues-- if llvmPackages_latest is what's receiving the bi-weekly releases, I don't think any of us individual maintainers have the bandwidth/compute to keep up. I don't, anyway. A hydra job(set), maybe rigged to some update bot helper, would be appropriate I think (although not my resources to allocate), especially for testing other platforms (+cross, (+musl, +static? 😇)). Back when I helped update this more regularly, that's basically what I was doing on my own hydra instance + builders (which I no longer have).

I don't know if we can test for it, but Mesa has had (subtle) breakages from LLVM updates before (IIRC some AMGPU thing manifesting as a rendering issue for users?), not that we need more pressure on us about this 😇 . (Unfortunately I think we'll just need our unstable users to help us catch those things, I think)

@dtzWill: What's your current status? IIRC I haven't seen your handles on LLVM PRs/issues often anymore but I could be wrong. Are you still interested in maintaining LLVM?

Apologies for not maintaining things more actively, I do use LLVM bits quite actively so I at least am good for kicking the tires. See my recent PR's unbreaking LLVM 14 bits after the quick bump (speaking of which LLDB still needs fixing, I have some WIP bits on PR's for this...). And FWIW a recent PR adding MLIR + flang (little rough). I also maintain the 'lit' package although TBH that's quite simple to keep up-to-date :).

Also: Maybe we want to make a “formal” llvm team.

If there's an LLVM team I'd like to be part of it. I was away for a few years and I'm unfortunately incredibly busy for the next month or two, but this is important to me in what's likely a long-term way :). I'll see what I can do, but I can't lead the charge.

[doyougnu]

+1 for documentation.

I'm currently trying to build llvm_15 for work (the new js-backend for GHC requires it) and running into a lot of patch issues which I do not understand where they come from, why they are needed or how to proceed. Honestly any documentation would be incredibly helpful. The same issues happen with llvm_latest after updating with either script as well.

[primeos]

the git conflict resolution seems like it may require deeper knowledge than I currently have. Could you put some steps down that I could try to follow to see if I have the expertise to do it?

It's not really difficult but requires (advanced?) VCS/Git knowledge. Unfortunately it'll probably take quite a bit of text to explain and I don't have a good link :o The way I do it (using Git) is basically: Checkout the old LLVM revision, import the Nixpkgs patches, rebase against the new LLVM revision (that way Git will perform the 3-way-merge for me and I just have to resolve the conflicts (and I use merge.conflictStyle = "diff3" in my ~/.gitconfig (s. man git-config) so that I always have all of the context to resolve the conflits). And if you keep up with Nixpkgs changes (i.e. keep the imported LLVM patches in sync) you only have to rebase next time. Anyway, that's just how I do it - there could be good or even better alternatives.

The GNU install dirs patches are difficult, I agree. The git stuff isn't so bad (although this last instance was rough) I think they're the biggest pain point re:getting things building (vetting/testing aside for now).

Yeah, it's just tedious work that takes time and I don't have the motivation for it anymore (it's important but I lack time for it and only need LLVM for Chromium).

Can we push on this?

AFAIK it's progressing slowly but I stopped following the progress (IIRC quite a few patches have been merged though). Anyway, it's progressing but not nearly as fast as I was hoping when they were added to Nixpkgs...

Or are there other options/approaches we can take? If that's not in the cards, how about we work on documenting what they do and why they're needed (or what to check to ensure things are as-expected) so that it's easier to port them with confidence?

@Ericson2314 could you give us a brief update on the current status/progress/plan of those GNU install dir patches?
re:Documentation: Most of the relevant information if probably linked here + in the commit message(s): #111487 (although we'd need something up-to-date... regarding the progress, Phabricator links, etc.).

If there's an LLVM team I'd like to be part of it.

Cool :) I think it'd be good/important to create such a team.

+1 for documentation.

Also +1 for that. IIRC we currently have basically no LLVM documentation at all :o

I'm currently trying to build llvm_15 for work (the new js-backend for GHC requires it) and running into a lot of patch issues which I do not understand where they come from, why they are needed or how to proceed. Honestly any documentation would be incredibly helpful. The same issues happen with llvm_latest after updating with either script as well.

Yeah, packaging newer LLVM releases or updating llvm_latest is basically a PITA atm (time consuming).
Another PITA is that PRs often don't update pkgs/development/compilers/llvm/git/ as well (which would be the most important thing to target...). Ideally, packaging a new LLVM release should require few changes other than cp -R pkgs/development/compilers/llvm/git pkgs/development/compilers/llvm/$n but in reality it requires digging through the Git history and applying missing patches first...

[Ericson2314]

@primeos I just landed a few more things upstream. The version 15 fork is going to happen soon I heard, so trying to shrink the patches a bit more before that.

That bottleneck is, as always, getting people to review the patches upstream. Once they do, they tend not to have any complaints.

[inclyc]

Hi Nixpkgs folks, I'd like to nominate myself and join the maintenance of LLVM related packages! I'm using NixOS now, and I can help with reviewing PRs & testing packages & upstream contacts. I'm working on the LLVM project (committer) and very happy to have such an elegant package management tool Nix! It would be nice if I could help a little bit :)

[inclyc]

Could we have a team and mentioned (subscribed) LLVM-related topics (?

We have a series GitHub teams at the llvm-project, and github notifies me if the PR/issue are related to this topic. (llvm/llvm-project#60066 (comment) for example)

[sternenseemann]

Could we have a team and mentioned (subscribed) LLVM-related topics (?

We can quite easily make a nixpkgs maintainer team #211401, creating a GH team in the NixOS org could be a second step if we feel it helps.

[RossComputerGuy]

We do have LLVM maintainers that improve the Nix expressions, handle PRs, etc. but we do severely lack maintainers to package new LLVM versions.

Well, I recently joined as a maintainer. I just found this issue, though I don't have the ability to commit I do have time and skills. I pretty much got started since LLVM 17. Do we need more than just one more maintainer? I can also create an LLVM team on GitHub since it seems there's a good number of us.

[RossComputerGuy]

I made @NixOS/llvm.

[silvanshade]

Well, I recently joined as a maintainer. I just found this issue, though I don't have the ability to commit I do have time and skills. I pretty much got started since LLVM 17. Do we need more than just one more maintainer? I can also create an LLVM team on GitHub since it seems there's a good number of us.

@RossComputerGuy I would like to contribute to this.

I'm still a bit new to Nix and NixOS but I've been working on a lot of toolchain related stuff with Nix lately, like custom derivations for stdenv, binutils, mold, LLVM, GCC, etc. (mostly for targeting RISC-V).

I also have some experience with the LLVM build system internals. Although it's a bit out of date now (mainly because I started using NixOS!), I worked on https://github.com/llvmup/toolchains which is a cmake superbuild project which is intended to make it easier to build and package LLVM toolchain components.

Please let me know how I can help.

[RossComputerGuy]

@silvanshade You can join the NixOS Matrix and then join the compilers team chat. That will be the best place to communicate without clogging up this issue.