mesa doesn't cross-compile anymore

[Mindavi]

Describe the bug

Mesa used to cross-compile fine, but it doesn't anymore. I have a feeling that this is an issue that might need to be discussed upstream, but I'd like to open this for visibility.

I hope someone can help me figure out why it can't find llvm anymore and if or how that can be resolved. Mesa is quite a big beast and it's not that easy to find where it regressed.

Steps To Reproduce

Steps to reproduce the behavior:

1. nix-build -A pkgsCross.aarch64-multiplatform.mesa

The build error

Run-time dependency libdrm_nouveau found: YES 2.4.107
Run-time dependency libdrm found: YES 2.4.107
WARNING: Ignoring LLVM CMake dependency because dynamic was requested
llvm-config found: NO need '>= 3.9.0'
Run-time dependency LLVM found: NO (tried cmake and config-tool)
Looking for a fallback subproject for the dependency llvm (modules: bitwriter, engine, mcdisassembler, mcjit, core, executionengine, scalaropts, transformutils, instcombine, native)
Neither a subproject directory nor a llvm.wrap file was found.
Subproject  llvm is buildable: NO (disabling)
Dependency llvm (modules: bitwriter, engine, mcdisassembler, mcjit, core, executionengine, scalaropts, transformutils, instcombine, native) from subproject subprojects/llvm found: NO (subproject failed to configure)

meson.build:1708:2: ERROR: Problem encountered: The following drivers require LLVM: Radv, RadeonSI, SWR, Lavapipe. One of these is enabled, but LLVM is disabled.

A full log can be found at /build/mesa-21.2.1/build/meson-logs/meson-log.txt
error: builder for '/nix/store/2i9d9l42wl268cbl9smkcnds4syc22vy-mesa-aarch64-unknown-linux-gnu-21.2.1.drv' failed with exit code 1;
       last 10 log lines:
       > llvm-config found: NO need '>= 3.9.0'
       > Run-time dependency LLVM found: NO (tried cmake and config-tool)
       > Looking for a fallback subproject for the dependency llvm (modules: bitwriter, engine, mcdisassembler, mcjit, core, executionengine, scalaropts, transformutils, instcombine, native)
       > Neither a subproject directory nor a llvm.wrap file was found.
       > Subproject  llvm is buildable: NO (disabling)
       > Dependency llvm (modules: bitwriter, engine, mcdisassembler, mcjit, core, executionengine, scalaropts, transformutils, instcombine, native) from subproject subprojects/llvm found: NO (subproject failed to configure)
       >
       > meson.build:1708:2: ERROR: Problem encountered: The following drivers require LLVM: Radv, RadeonSI, SWR, Lavapipe. One of these is enabled, but LLVM is disabled.
       >
       > A full log can be found at /build/mesa-21.2.1/build/meson-logs/meson-log.txt
       For full logs, run 'nix log /nix/store/2i9d9l42wl268cbl9smkcnds4syc22vy-mesa-aarch64-unknown-linux-gnu-21.2.1.drv'.
error: build of '/nix/store/6f2gb9394p00psh74cajzkqqy150zwvf-mesa-aarch64-unknown-linux-gnu-21.2.1.drv' failed

Expected behavior

Mesa cross-compiles properly and can find llvm.

Additional context

I figured out mesa broke between the change from 21.0.x to 21.1.x. Changing with which llvm version mesa builds does not seem to affect anything (I tried llvm_11). Removing the llvmPackages_latest override on top-level doesn't seem to affect anything.

Notify maintainers

@primeos @vcunat

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

 - system: `"x86_64-linux"`
 - host os: `Linux 5.13.12, NixOS, 21.11 (Porcupine)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.4pre20210802_47e96bb`
 - channels(root): `"nixos-21.11pre312125.21c937f8cb1, nixpkgs-21.11pre303315.16105403bdd"`
 - channels(rick): `""`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute: pkgsCross.aarch64-multiplatform.mesa

[vcunat]

I tried on current master (2e30d4c) but llvm wouldn't build already:

/nix/store/v8imx1nvyz0hgvx9cbcmh6gp4ngw3ffj-binutils-2.35.1/bin/ld: /nix/store/ysi7nyk16g05mzghpp0c5f9iwknf8mw
n-ncurses-6.2-aarch64-unknown-linux-gnu/lib/libtinfo.so: error adding symbols: file in wrong format
collect2: error: ld returned 1 exit status

builder for '/nix/store/ahy0bpa060gyy97wzl1kmlrxk1nd6dwg-llvm-aarch64-unknown-linux-gnu-12.0.1.drv' failed

[Mindavi]

Oh yeah, sorry. This may help: #136923 or downgrading to llvm11 for llvm_latest.

[vcunat]

With that llvm-config is an aarch64 executable, so it can't work when cross-compiling from x86_64.

[Mindavi]

llvm-config-native is an x86_64 executable though, I think that one should be used. And this worked with an older version of mesa, so I still think it's something that changed on mesa's side, not llvm's side. I'l try with the older mesa version to see if that builds with llvm_12 (with the patch I made to llvm).

[vcunat]

Anyway, meson log doesn't seem very helpful:

WARNING: Ignoring LLVM CMake dependency because dynamic was requested
llvm-config binary missing from cross or native file, or env var undefined.
Default target is not allowed for cross use
llvm-config found: NO need '>= 3.9.0'

[Mindavi]

I tried building an older version of mesa with llvm_12, and that seemed to work fine. However, it seems like it doesn't do anything with llvm, so this might have been a problem for longer already. The only difference is that it doesn't build at all now instead of ignoring that llvm cannot be found.

The commits I reverted to test this

8761186f0a69b8aa90a034d66386ae3eb21f3886
f5db95a96aef57c945d4741c2134104da1026b7d
364bb239ab417d7b06556812643e265fc5fe81fa
1b3ba289b2a03748d0a8b790c82b638b73a7dd01
b5a7a474d17a5a851c80a144fc4ea013079a4770
e23145b62ad582c954c6a43eb167147fe9882333
ecbe6c123f23aa72cb0ead618a8b2d3099a9b361
f885e987ef7bae172699ece3ef37c4473d88bd5a
afe09e41df7e59619340ccf1dfbab940adc1e3e3
012a33b0ded655184819e2270bd39e85c4d93cfd
70029711d476db16abd90ddb25bafe93942019b8
e56bed6bdb14416b37ed8557c70020a48a21e86f

So maybe the way to solve this is to disable building with llvm at all, or fixing it up properly so e.g. llvm-config-native is used. I don't know if it's still possible to build without llvm and what we miss in that case though.

[vcunat]

The following drivers require LLVM: Radv, RadeonSI, SWR, Lavapipe. One of these is enabled, but LLVM is disabled.

[vcunat]

Maybe a "native file" which specifies the path to llvm-config-native, I don't know: https://docs.mesa3d.org/meson.html#llvm

[Mindavi]

Ooh, that might be a good one. I'll have to look into that. I totally missed that documentation.

[Mindavi]

This seems to work:

commit befd38c612fc8ae7fde35350c4f5ce0d0876112b (HEAD -> pinephone-patches-3)
Author: Rick van Schijndel <rol3517@gmail.com>
Date:   Wed Sep 8 08:25:02 2021 +0200

    mesa: fix cross-compilation by properly using llvm

diff --git a/pkgs/development/libraries/mesa/default.nix b/pkgs/development/libraries/mesa/default.nix
index 62e95dc8e99..1e752e448c5 100644
--- a/pkgs/development/libraries/mesa/default.nix
+++ b/pkgs/development/libraries/mesa/default.nix
@@ -13,6 +13,7 @@
 , withValgrind ? !stdenv.isDarwin && lib.meta.availableOn stdenv.hostPlatform valgrind-light, valgrind-light
 , enableGalliumNine ? stdenv.isLinux
 , enableOSMesa ? stdenv.isLinux
+, writeText
 }:
 
 /** Packaging design:
@@ -118,6 +119,14 @@ self = stdenv.mkDerivation {
     "-Dmicrosoft-clc=disabled" # Only relevant on Windows (OpenCL 1.2 API on top of D3D12)
   ] ++ optionals stdenv.isLinux [
     "-Dglvnd=true"
+  ] ++ optionals (stdenv.buildPlatform != stdenv.hostPlatform) [
+    # Create meson cross file.
+    # See https://docs.mesa3d.org/meson.html#llvm.
+    # This appears to not override the default cross-file that's generated by nixpkgs.
+    "--cross-file=${writeText "cross-file" ''
+      [binaries]
+      llvm-config = '${llvmPackages.llvm.dev}/bin/llvm-config-native'
+    ''}"
   ];
 
   buildInputs = with xorg; [

I'll upstream this later, but anyone is free to do that before I'll do it too.

Thanks for the hint Vladimir!

[vcunat]

Actually I wonder why meson doesn't use the -native suffix automatically, as they seem to have a built-in detection engine for llvm in particular: https://mesonbuild.com/Dependencies.html#llvm

[Mindavi]

They do have detection, but it might be that the -native suffix is a nixpkgs thing. I'm not sure though. I don't know if it's defined somewhere else as well, where meson should pick it up?

[alyssais]

A cross file might be the way to go, but Mesa is just using the standard Meson LLVM detection stuff, right? If so, we'd want this to be fixed using our normal cross file, so it worked for all packages, not just mesa, right?

[Mindavi]

A more generic solution is preferable, not sure how that'd work though. We can't just add llvm to the cross file in general since that would add llvm in the closure for everything. Maybe some hook could be added when llvm is added as a package, that might work? I don't know how that works though.

[alyssais]

I was able to make some progress with the following patch, which solves the problem in a way that doesn't require changing the cross file in every Meson-using package. There are still a couple of problems with it though. It gets a bit further, and then fails with meson.build:1643:6: ERROR: Problem encountered: The Nouveau driver requires rtti. You either need to turn off nouveau or use an LLVM built with LLVM_ENABLE_RTTI. But our LLVM is built with LLVM_ENABLE_RTTI. llvm-config-native --has-rtti does indeed print "NO". So probably something else needs to be fixed in LLVM? The other problem is that llvmPackages in the mesa derivation refers to the llvmPackages_12 that's manually passed in by all-packages.nix. So if I were to refer to pkgsHostHost.libllvm.llvm in nativeBuildInputs, I'd get the default version of LLVM, instead of the version being used in the rest of the derivation. But if I were to just use llvmPackages like in this patch, cross-compilation will break as soon as that override is removed, because then splicing will come into play. I'm not sure if we have an idiom for dealing with that? (cc @Ericson2314 @sternenseemann I guess) diff --git i/pkgs/development/libraries/mesa/default.nix w/pkgs/development/libraries/mesa/default.nix index 876a3c015a0..e069c3786e0 100644 --- i/pkgs/development/libraries/mesa/default.nix +++ w/pkgs/development/libraries/mesa/default.nix @@ -142,6 +142,7 @@ self = stdenv.mkDerivation { meson pkg-config ninja intltool bison flex file python3Packages.python python3Packages.Mako + (lib.getDev llvmPackages.libllvm) ] ++ lib.optionals (elem "wayland" eglPlatforms) [ wayland-scanner ]; diff --git i/pkgs/stdenv/generic/make-derivation.nix w/pkgs/stdenv/generic/make-derivation.nix index 56cfa0c503f..8fa30637049 100644 --- i/pkgs/stdenv/generic/make-derivation.nix +++ w/pkgs/stdenv/generic/make-derivation.nix @@ -305,6 +305,9 @@ else let cpu_family = '${cpuFamily stdenv.targetPlatform}' cpu = '${stdenv.targetPlatform.parsed.cpu.name}' endian = ${if stdenv.targetPlatform.isLittleEndian then "'little'" else "'big'"} + + [binaries] + llvm-config = 'llvm-config-native' ''; in [ "--cross-file=${crossFile}" ] ++ mesonFlags; } // lib.optionalAttrs (attrs.enableParallelBuilding or false) {

[sternenseemann]

But our LLVM is built with LLVM_ENABLE_RTTI. llvm-config-native --has-rtti
does indeed print "NO". So probably something else needs to be fixed in
LLVM?

Wondering if this may be a similar issue to #127505.

[Mindavi]

I don't understand why that happens with the generic solution, but not with my patch 🧐

[Ericson2314]

diff --git i/pkgs/stdenv/generic/make-derivation.nix w/pkgs/stdenv/generic/make-derivation.nix
index 56cfa0c503f..8fa30637049 100644
--- i/pkgs/stdenv/generic/make-derivation.nix
+++ w/pkgs/stdenv/generic/make-derivation.nix
@@ -305,6 +305,9 @@ else let
           cpu_family = '${cpuFamily stdenv.targetPlatform}'
           cpu = '${stdenv.targetPlatform.parsed.cpu.name}'
           endian = ${if stdenv.targetPlatform.isLittleEndian then "'little'" else "'big'"}
+
+          [binaries]
+          llvm-config = 'llvm-config-native'
         '';
       in [ "--cross-file=${crossFile}" ] ++ mesonFlags;
     } // lib.optionalAttrs (attrs.enableParallelBuilding or false) {

perhaps we should start with that? Seems necessary if not sufficient.

[Mindavi]

The comment from @Ericson2314 + adding llvmPackages.llvm to nativeBuildInputs seems to work (at least to make meson happy). Now building...

Edit: It built!