Impure path used during linking of unikernel image

[mazunki]

Attempting to build example/ (through nix-shell ./shell.nix) on Gentoo gives the following errors:

ld.lld: error: /nix/store/wkn6ivpdjri06sjig43k43n72cidzb4p-includeos-static-x86_64-unknown-linux-musl-dev/libcxx/lib/libc++.a(cxa_exception_storage.cpp.o): unable to find library from dependent library specifier: pthread
ld.lld: error: /nix/store/wkn6ivpdjri06sjig43k43n72cidzb4p-includeos-static-x86_64-unknown-linux-musl-dev/libcxx/lib/libc++.a(fallback_malloc.cpp.o): unable to find library from dependent library specifier: pthread
ld.lld: error: /nix/store/wkn6ivpdjri06sjig43k43n72cidzb4p-includeos-static-x86_64-unknown-linux-musl-dev/libcxx/lib/libc++.a(memory.cpp.o): unable to find library from dependent library specifier: pthread
ld.lld: error: /nix/store/wkn6ivpdjri06sjig43k43n72cidzb4p-includeos-static-x86_64-unknown-linux-musl-dev/libcxx/lib/libc++.a(mutex.cpp.o): unable to find library from dependent library specifier: pthread
ld.lld: error: /nix/store/wkn6ivpdjri06sjig43k43n72cidzb4p-includeos-static-x86_64-unknown-linux-musl-dev/libcxx/lib/libc++.a(cxa_guard.cpp.o): unable to find library from dependent library specifier: pthread
ld.lld: error: /nix/store/wkn6ivpdjri06sjig43k43n72cidzb4p-includeos-static-x86_64-unknown-linux-musl-dev/libcxx/lib/libc++.a(condition_variable.cpp.o): unable to find library from dependent library specifier: pthread
ld.lld: error: /nix/store/wkn6ivpdjri06sjig43k43n72cidzb4p-includeos-static-x86_64-unknown-linux-musl-dev/libcxx/lib/libc++.a(thread.cpp.o): unable to find library from dependent library specifier: pthread
ld.lld: error: /nix/store/wkn6ivpdjri06sjig43k43n72cidzb4p-includeos-static-x86_64-unknown-linux-musl-dev/libunwind/lib/libunwind.a(libunwind.cpp.o): unable to find library from dependent library specifier: dl
ld.lld: error: /nix/store/wkn6ivpdjri06sjig43k43n72cidzb4p-includeos-static-x86_64-unknown-linux-musl-dev/libunwind/lib/libunwind.a(libunwind.cpp.o): unable to find library from dependent library specifier: pthread

On other projects adding set(CMAKE_LINKER_TYPE LLD) to example/CMakeLists.txt fixes this issue (is this the correct solution?), but here this raises another warning: ld.lld: error: unknown argument '-fuse-ld=lld'.

Instead, I can add -DCMAKE_LINKER="$(command -v ld)" to shell.nix, which seems to fix it, but that feels like a dirty hack:

diff --git a/shell.nix b/shell.nix
index b26dbef07..ad3075213 100644
--- a/shell.nix
+++ b/shell.nix
@@ -64,7 +64,7 @@ includeos.pkgs.mkShell.override { inherit (includeos) stdenv; } rec {
         pushd "$BUILDPATH"
     fi
     cmake "$unikernel" -DARCH=x86_64 -DINCLUDEOS_PACKAGE=${includeos} -DCMAKE_MODULE_PATH=${includeos}/cmake \
-                     -DFOR_PRODUCTION=OFF
+                     -DFOR_PRODUCTION=OFF -DCMAKE_LINKER="$(command -v ld)"
     make -j $NIX_BUILD_CORES
     echo -e "\n====================== IncludeOS nix-shell ====================="
     if [ -z "${buildpath}" ]; then

Why is Nix picking an impure linker (i.e. /usr/lib/llvm/20/bin/ld.lld) unless explicitly telling it to use LLD, despite running nix-shell --pure? Is there a better solution than hardcoding the linker path through the cmake call?

None of this is necessary on NixOS. I haven't tested on other non-Gentoo systems, but I would imagine I'm not alone with this issue, and I feel like we should prevent this issue from our side.

[fwsGonzo]

set(CMAKE_LINKER_TYPE LLD) is new to me at least. Looks like it was added very recently, 3.29 to be specific. Everyone suggests the standard -fuse-ld=lld but that doesn't work when you have versioned clang either.

if (USE_MOLD)
    set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} --ld-path=/usr/local/bin/mold")
elseif (USE_LLD)
	set(CMAKE_AR "llvm-ar-${CLANG_VERSION}")
	set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -fuse-ld=lld-${CLANG_VERSION}")
endif()

I have this in another project, which I don't recommend using. But it shows I had a problem with selecting lld as well.

[mazunki]

Yeah, I believe set(CMAKE_LINKER_TYPE LLD) basically enables the CMAKE_EXE_LINKER_FLAGS for executables (and CMAKE_SHARED_LINKER_FLAGS for libraries, and so on). I might be wrong on the details of this.

On my system I don't have llvm-ar-21 as an executable, nor does ${CLANG_VERSION} have a value. Hardcoding the linker to llvm-ar with -fuse-ld=lld-21 or -fuse-ld=lld both report an unknown argument. I imagine your reason for not recommending it is due to these mobile interactions?

The interesting part to me is that the set(CMAKE_LINKER_TYPE LLD) works perfectly well on another unikernel. I should be able to bisect what's going on there, but it might take a while.

[mazunki]

Looks like it was added very recently, 3.29 to be specific.

Thanks, I realized the reason the option wasn't working was because of the version pinning in the CMake files.

[alfreb]

This is surprising- I thought we had never built with LLVM's linker, except in some experiments to test LTO.

alfreb@testctrl:~/IncludeOS$ nix-shell
Attempting to build unikernel: 
/home/alfreb/IncludeOS/example
...
[100%] Linking CXX executable hello_includeos.elf.bin
/nix/store/xi6fvkmaq01ggn3zqgczb6vd01xl02kc-binutils-static-x86_64-unknown-linux-musl-2.44/bin/ld: warning: __syscall_entry.asm.o: missing .note.GNU-stack section implies executable stack
/nix/store/xi6fvkmaq01ggn3zqgczb6vd01xl02kc-binutils-static-x86_64-unknown-linux-musl-2.44/bin/ld: NOTE: This behaviour is deprecated and will be removed in a future version of the linker
/nix/store/xi6fvkmaq01ggn3zqgczb6vd01xl02kc-binutils-static-x86_64-unknown-linux-musl-2.44/bin/ld: warning: hello_includeos.elf.bin has a LOAD segment with RWX permissions

So the linker being used is /nix/store/xi6fvkmaq01ggn3zqgczb6vd01xl02kc-binutils-static-x86_64-unknown-linux-musl-2.44/bin/ld.

Inside the shell we can inspect this:

[nix-shell:/run/user/1001/tmp.3tApbd2GRo]$ ls -lah  /nix/store/xi6fvkmaq01ggn3zqgczb6vd01xl02kc-binutils-static-x86_64-unknown-linux-musl-2.44/bin/ld
lrwxrwxrwx 1 root root 6 Jan  1  1970 /nix/store/xi6fvkmaq01ggn3zqgczb6vd01xl02kc-binutils-static-x86_64-unknown-linux-musl-2.44/bin/ld -> ld.bfd

Here ld.bfd is a magic env var created by nixpkgs (and this is where me and nixpkgs don't see eye to eye as this is very hairy, but I don't know of anything better out there).

$ file $(which ld.bfd)
/nix/store/sad5lh8sylfqdfks86vrfil9k1s9n692-clang-wrapper-static-18.1.8/bin/ld.bfd: symbolic link to /nix/store/zw0liivgvxfmxd5k17g0gzrql39pr6ib-binutils-wrapper-static-2.44/bin/ld.bfd
$ file /nix/store/zw0liivgvxfmxd5k17g0gzrql39pr6ib-binutils-wrapper-static-2.44/bin/ld.bfd
/nix/store/zw0liivgvxfmxd5k17g0gzrql39pr6ib-binutils-wrapper-static-2.44/bin/ld.bfd: a /nix/store/h7a019l5srm5p9307mv6k0y6arnsfkha-bash-static-x86_64-unknown-linux-musl-5.2p37/bin/bash script, ASCII text executable

So that's a symlink to a bash script, which wraps the linker which is the default for this toolchain (I think that's determined by llvmPackages). Calling that script will invoke the linker we actually used:

$ /nix/store/zw0liivgvxfmxd5k17g0gzrql39pr6ib-binutils-wrapper-static-2.44/bin/ld.bfd --version
GNU ld (GNU Binutils) 2.44
Copyright (C) 2025 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or (at your option) a later version.
This program has absolutely no warranty.

I see the flags mentioned here seems to indicate we're using the USE_LLD CMake flag, which would indeed override this, but it seems to be default to off:

alfreb@testctrl:~/IncludeOS$ grep -r USE_LLD -A 3
cmake/linux.service.cmake:option(USE_LLD "Allow linking against LTO archives" OFF)
cmake/linux.service.cmake-
cmake/linux.service.cmake-if(DEBUGGING)
cmake/linux.service.cmake-  set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -g -O0")
--
cmake/linux.service.cmake:  if (ENABLE_LTO OR USE_LLD)
cmake/linux.service.cmake-    set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -fuse-ld=lld")
cmake/linux.service.cmake-  endif()
cmake/linux.service.cmake-endif()

Am I missing something?

[mazunki]

This is surprising- I thought we had never built with LLVM's linker, except in some experiments to test LTO.

I think that's correct. I believe we default to ld.bfd. My question in this regard is mostly whether that's something we want, given that we're using LLVM for everything else.

Am I missing something?

Interesting observation:

$ echo ${PATH} | tr : \\n
/home/maz/.nix-profile/bin
/usr/local/sbin
/usr/local/bin
/usr/sbin
/usr/bin
/sbin
/bin
/opt/bin
/usr/lib/llvm/21/bin
/usr/lib/llvm/20/bin
/usr/lib/llvm/19/bin
/home/maz/.local/src/shellscripts
$ nix-shell
# ... fails

despite which ld in the shell showing /nix/store/sad5lh8sylfqdfks86vrfil9k1s9n692-clang-wrapper-static-18.1.8/bin/ld (same as in a pure shell).

while the following works correctly:

$ env \
    PATH="/home/maz/.nix-profile/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin:/home/maz/.local/src/shellscripts" \
    nix-shell

While I could blame it on Gentoo for exporting these PATH entries, I feel like nix should be prioritizing its own version of ld during linking, given that ~/.nix-profile/bin is first in PATH here. Could it be CMake is looking for ld.lld in PATH before it checks for ld, even when LLD wasn't requested?

[alfreb]

Right so about LLD: Ideally we would be able to compile and run with both GCC stack and LLVM stack, as building with more compilers can uncover more issues and increase compatibility. But since we don't have builds in CI atm. I don't think we should prioritize it. I think keeping it simple and easy to reason about is more important. And yea, there are plenty of nix quirks (or nixpkgs quirks to be fair) that are not helping. My mental model is that we should try to use what ever is the default in the nix toolchain we have. If there are concrete benefits to switching to LLD I'm happy to consider that, but then let's make that a separate issue.

Then about impure paths; I'm not sure I follow - are you saying nix-shell is the command that fails for you? For me I don't get any impure paths in $PATH when running nix-shell --pure, but I do when running just nix-shell.

As for which linker should be prioritized; yes I would expect nixpkgs to provide the linker, in our case I think its GNU ld:

nix repl --expr 'with import <nixpkgs> {}; llvmPackages_18.stdenv.cc.bintools.bintools'
«derivation /nix/store/52zs1y9f4vf0vpvgid1jq41l1asrqw56-binutils-wrapper-2.41.drv»

So that's from binutils. I'm not sure there's really an IncludeOS specific issue here - if it is, please help me understand, otherwise let's close this one for now. Feel free to open a separate issue for trying to link with LLD to get LTO if you'd like to see work on that.

[mazunki]

If there are concrete benefits to switching to LLD I'm happy to consider that, but then let's make that a separate issue.

That's fair. I've updated the PR to drop this change: the CMake version currently chosen is deprecated, and gives warnings.

I'm not sure there's really an IncludeOS specific issue here - if it is, please help me understand, otherwise let's close this one for now.

I don't seem to get any issues with --pure paths anymore. Either I misreported that, or something has changed on my system. Nonetheless, nix-shell to build the example shell is not working.

I made this small edit in the shell.nix

    echo $PATH | tr : '\n'
    printf '\n\n====\n\n'
    export PATH="$(printf %s "$PATH" | sed -e 's;\(^\|:\)/usr/lib/llvm/[0-9][0-9]*/bin;;g')"
    echo $PATH | tr : '\n'
    exit
    cmake "$unikernel" -DARCH=x86_64 -DINCLUDEOS_PACKAGE=${includeos} -DCMAKE_MODULE_PATH=${includeos}/cmake \
                     -DFOR_PRODUCTION=OFF

which gives the following output:

Attempting to build unikernel: 
/home/maz/repos/includeos/IncludeOS/example
/tmp/nix-shell-13805-2851616970/tmp.xJFSJ6WUre ~/repos/includeos/IncludeOS
/nix/store/ddx7976jyll30xjbasghv9jailswprcp-bash-interactive-5.3p3/bin
/nix/store/00ylhcgpgh7rmxqzh76yhhjhhqfjf31k-python3-3.12.10-env/bin
/nix/store/wphngc22a7aphbp5pi5jqmqlsqmisgn5-cmake-3.31.6/bin
/nix/store/p5g6x49478mzfqy1mvj7b1hvdzsd0cqi-nasm-2.16.03/bin
/nix/store/wm175cabr53vnkmlrb8y6bbqcv4bj5cv-qemu-9.2.3/bin
/nix/store/dqcl4f3r1z7ck24rh9dw2i6506g7wky5-which-2.23/bin
/nix/store/9318hbk3akgqjkx5yn3b8q8rfdycqwdj-grub-2.12/bin
/nix/store/9hpylx077slqmzb5pz8818mxjws3appp-iputils-20240905/bin
/nix/store/1191qk37q1bxyj43j0y1l534jvsckyma-acl-2.3.2-bin/bin
/nix/store/jn32qcr1j7s1ppw1in61iyyn0vagkwjj-libisoburn-1.5.6/bin
/nix/store/2jc1jmzis19adawjwhl8qhdvh7vlbk0q-patchelf-0.15.0/bin
/nix/store/sad5lh8sylfqdfks86vrfil9k1s9n692-clang-wrapper-static-18.1.8/bin
/nix/store/ajwbvlkqrq98b9k0k2m51lq1zmar0iyc-clang-static-x86_64-unknown-linux-musl-18.1.8/bin
/nix/store/hz69hi4q485r1lfzmdi0l270lp04icsz-coreutils-static-x86_64-unknown-linux-musl-9.7/bin
/nix/store/zw0liivgvxfmxd5k17g0gzrql39pr6ib-binutils-wrapper-static-2.44/bin
/nix/store/xi6fvkmaq01ggn3zqgczb6vd01xl02kc-binutils-static-x86_64-unknown-linux-musl-2.44/bin
/nix/store/87fck6hm17chxjq7badb11mq036zbyv9-coreutils-9.7/bin
/nix/store/7y59hzi3svdj1xjddjn2k7km96pifcyl-findutils-4.10.0/bin
/nix/store/7h0sard22wnbz0jyz07w8y9y0fcs795r-diffutils-3.12/bin
/nix/store/clbb2cvigynr235ab5zgi18dyavznlk2-gnused-4.9/bin
/nix/store/gqmr3gixlddz3667ba1iyqck3c0dkpvd-gnugrep-3.11/bin
/nix/store/fcyn0dqszgfysiasdmkv1jh3syncajay-gawk-5.3.2/bin
/nix/store/wrxvqj822kz8746608lgns7h8mkpn79f-gnutar-1.35/bin
/nix/store/afhkqb5a94zlwjxigsnwsfwkf38h21dk-gzip-1.14/bin
/nix/store/1abbyfv3bpxalfjfgpmwg8jcy931bf76-bzip2-1.0.8-bin/bin
/nix/store/kv10h4pidkmx8cjs2sw2pi9rlcnighbc-gnumake-4.4.1/bin
/nix/store/xy4jjgw87sbgwylm5kn047d9gkbhsr9x-bash-5.2p37/bin
/nix/store/x0kaspzb5jqvgp357bj27z6iq24ximfg-patch-2.7.6/bin
/nix/store/98zamhd8d0jq3skqwz28dlgph94mrqir-xz-5.8.1-bin/bin
/nix/store/imhzyxqr7swq08ip81az5kfa07r07kg0-file-5.46/bin
/home/maz/.nix-profile/bin
/usr/local/sbin
/usr/local/bin
/usr/sbin
/usr/bin
/sbin
/bin
/opt/bin
/usr/lib/llvm/21/bin
/usr/lib/llvm/20/bin
/usr/lib/llvm/19/bin
/home/maz/.local/src/shellscripts

====
/nix/store/ddx7976jyll30xjbasghv9jailswprcp-bash-interactive-5.3p3/bin
/nix/store/00ylhcgpgh7rmxqzh76yhhjhhqfjf31k-python3-3.12.10-env/bin
/nix/store/wphngc22a7aphbp5pi5jqmqlsqmisgn5-cmake-3.31.6/bin
/nix/store/p5g6x49478mzfqy1mvj7b1hvdzsd0cqi-nasm-2.16.03/bin
/nix/store/wm175cabr53vnkmlrb8y6bbqcv4bj5cv-qemu-9.2.3/bin
/nix/store/dqcl4f3r1z7ck24rh9dw2i6506g7wky5-which-2.23/bin
/nix/store/9318hbk3akgqjkx5yn3b8q8rfdycqwdj-grub-2.12/bin
/nix/store/9hpylx077slqmzb5pz8818mxjws3appp-iputils-20240905/bin
/nix/store/1191qk37q1bxyj43j0y1l534jvsckyma-acl-2.3.2-bin/bin
/nix/store/jn32qcr1j7s1ppw1in61iyyn0vagkwjj-libisoburn-1.5.6/bin
/nix/store/2jc1jmzis19adawjwhl8qhdvh7vlbk0q-patchelf-0.15.0/bin
/nix/store/sad5lh8sylfqdfks86vrfil9k1s9n692-clang-wrapper-static-18.1.8/bin
/nix/store/ajwbvlkqrq98b9k0k2m51lq1zmar0iyc-clang-static-x86_64-unknown-linux-musl-18.1.8/bin
/nix/store/hz69hi4q485r1lfzmdi0l270lp04icsz-coreutils-static-x86_64-unknown-linux-musl-9.7/bin
/nix/store/zw0liivgvxfmxd5k17g0gzrql39pr6ib-binutils-wrapper-static-2.44/bin
/nix/store/xi6fvkmaq01ggn3zqgczb6vd01xl02kc-binutils-static-x86_64-unknown-linux-musl-2.44/bin
/nix/store/87fck6hm17chxjq7badb11mq036zbyv9-coreutils-9.7/bin
/nix/store/7y59hzi3svdj1xjddjn2k7km96pifcyl-findutils-4.10.0/bin
/nix/store/7h0sard22wnbz0jyz07w8y9y0fcs795r-diffutils-3.12/bin
/nix/store/clbb2cvigynr235ab5zgi18dyavznlk2-gnused-4.9/bin
/nix/store/gqmr3gixlddz3667ba1iyqck3c0dkpvd-gnugrep-3.11/bin
/nix/store/fcyn0dqszgfysiasdmkv1jh3syncajay-gawk-5.3.2/bin
/nix/store/wrxvqj822kz8746608lgns7h8mkpn79f-gnutar-1.35/bin
/nix/store/afhkqb5a94zlwjxigsnwsfwkf38h21dk-gzip-1.14/bin
/nix/store/1abbyfv3bpxalfjfgpmwg8jcy931bf76-bzip2-1.0.8-bin/bin
/nix/store/kv10h4pidkmx8cjs2sw2pi9rlcnighbc-gnumake-4.4.1/bin
/nix/store/xy4jjgw87sbgwylm5kn047d9gkbhsr9x-bash-5.2p37/bin
/nix/store/x0kaspzb5jqvgp357bj27z6iq24ximfg-patch-2.7.6/bin
/nix/store/98zamhd8d0jq3skqwz28dlgph94mrqir-xz-5.8.1-bin/bin
/nix/store/imhzyxqr7swq08ip81az5kfa07r07kg0-file-5.46/bin
/home/maz/.nix-profile/bin
/usr/local/sbin
/usr/local/bin
/usr/sbin
/usr/bin
/sbin
/bin
/opt/bin
/home/maz/.local/src/shellscripts

The whole reason this is an issue is because we're building the example unikernel from the shellHook, which uses an impure PATH by design. That said, I fail to understand why this is not working. After the updated PATH, it works and links fine.
Maybe we should just move building of the unikernel to example.nix instead, instead of putting it in shellHook?