Add nixos support

[lromor]

Hi!

I would like to work on adding support to nixos. Toghether with this introductory PR, I have a quick question. On of the issues I have now when using the latest toolchain is:

	File "/home/lromor/.cache/bazel/_bazel_lromor/38bd39a93dc2b485b8f35263fd6e2b00/external/rules_cc/cc/find_cc_toolchain.bzl", line 68, column 13, in find_cc_toolchain
		fail("In order to use find_cc_toolchain, your rule has to depend on C++ toolchain. See find_cc_toolchain.bzl docs for details.")
Error in fail: In order to use find_cc_toolchain, your rule has to depend on C++ toolchain. See find_cc_toolchain.bzl docs for details.

Do tou have any clues on how to fix this? Is it something fixable in the repo or does it require I wrap cc_* rules with my own that uses the default_toolcain() function?

[rrbutani]

@lromor Hi! Can you share the example workspace you're using or provide a little more context about the target you're building that leads to the error? I'm a little confused; the error seems unrelated to bazel-toolchain. Also, just to be sure:

• Does the target you're building work without bazel-toolchain but with --incompatible_enable_cc_toolchain_resolution?
• Do the targets being built come from rules_cc rules or custom rules?

[rrbutani]

re NixOS support: this is something I'd very much like to see in bazel-toolchain! Unfortunately I don't think it'll be as simple as modifying the distro selection logic.

The LLVM releases that bazel-toolchain uses are dynamically linked against libraries like zlib and tinfo and libstdc++; because nixpkgs/NixOS don't materialize such libraries at system-wide loader-accessible paths like /lib (and instead embed the nix store paths to such libraries in RUNPATH) we can't run these binaries without modification or other setup:

[nixos@nixos:~/clang+llvm-15.0.6-aarch64-linux-gnu]$ ldd bin/clang
	linux-vdso.so.1 (0x0000ffff9ba4c000)
	libpthread.so.0 => /nix/store/g14swv4f0rg83naqj87g1mjmyvrmrxmf-glibc-2.35-163/lib/libpthread.so.0 (0x0000ffff92f10000)
	librt.so.1 => /nix/store/g14swv4f0rg83naqj87g1mjmyvrmrxmf-glibc-2.35-163/lib/librt.so.1 (0x0000ffff92ee0000)
	libdl.so.2 => /nix/store/g14swv4f0rg83naqj87g1mjmyvrmrxmf-glibc-2.35-163/lib/libdl.so.2 (0x0000ffff92eb0000)
	libm.so.6 => /nix/store/g14swv4f0rg83naqj87g1mjmyvrmrxmf-glibc-2.35-163/lib/libm.so.6 (0x0000ffff92e00000)
	libz.so.1 => not found
	libtinfo.so.6 => not found
	libstdc++.so.6 => not found
	libgcc_s.so.1 => /nix/store/g14swv4f0rg83naqj87g1mjmyvrmrxmf-glibc-2.35-163/lib/libgcc_s.so.1 (0x0000ffff92dd0000)
	libc.so.6 => /nix/store/g14swv4f0rg83naqj87g1mjmyvrmrxmf-glibc-2.35-163/lib/libc.so.6 (0x0000ffff92c20000)
	/lib/ld-linux-aarch64.so.1 => /nix/store/g14swv4f0rg83naqj87g1mjmyvrmrxmf-glibc-2.35-163/lib/ld-linux-aarch64.so.1 (0x0000ffff9ba0f000)

[nixos@nixos:~/clang+llvm-15.0.6-aarch64-linux-gnu]$ !$
bin/clang
-bash: bin/clang: No such file or directory

(note that some of the libraries in the above are resolved because NixOS preloads some libraries in the ld so cache)

---

Ultimately this means that to use upstream LLVM releases on NixOS, I think our options are:

• 1 patch the binaries
  • this would involve us, in the repo rule, grabbing patchelf and the necessary library deps from nixpkgs and patching the downloaded LLVM binaries
• 2 require that users of bazel-toolchain on NixOS wrap bazel in something like buildFHSUserEnv with the appropriate library deps
  • this sets up a chroot where libraries are mounted in at the usual loader-accessible locations
  • we wouldn't want to wrap just the LLVM binaries (because then you have to pay the setup overhead on every invocation) but wrapping bazel might be problematic too; I'm not sure if this interacts well with Bazel's sandbox
  • this is also probably not the best solution from a user standpoint; we'd want to add checks in the repo rule nudging users on NixOS towards the FHS wrappers (otherwise they'd just get inscrutable errors about LLVM binaries not being found)
• 3 use something like nix-ld or nix-autobahn
  • setting up NIX_LD and NIX_LD_LIBRARY_PATH is conceivably something we could have bazel-toolchain do (including grabbing libraries from nixpkgs and setting up a gcroot) but users would still need to enable nix-ld in their NixOS config

Alternatively we could look into procuring LLVM releases for NixOS from other sources:

• 4 perhaps statically linked binaries that we produce?
• 5 or from the llvmPackage_* sets in nixpkgs
  • we could fetch these from cache.nixos.org directly but that'd still involve us, in a repo rule, going and running nix commands
  • at that point it may just be better to direct users towards rules_nixpkgs's nixpkgs_cc_configure

I think the first option is the least bad in terms of user experience but it is complex. The third option is the simplest to implement (just distro checking and some docs and error messages) and is perhaps in line with how NixOS users use bazel anyways (i.e. under buildFHSUserEnv).

---

I'm willing to help implement and test any of the above but I think @siddharthab should chime in with what's actually in scope for bazel-toolchain and how much complexity the project is willing to take on to support NixOS.

[lromor]

@rrbutani

@lromor Hi! Can you share the example workspace you're using or provide a little more context about the target you're building that leads to the error? I'm a little confused; the error seems unrelated to bazel-toolchain.

Sure thing: I'm also a bit confused, I just used the README quickstart:

load("@com_grail_bazel_toolchain//toolchain:deps.bzl", "bazel_toolchain_dependencies")

bazel_toolchain_dependencies()

load("@com_grail_bazel_toolchain//toolchain:rules.bzl", "llvm_toolchain")

llvm_toolchain(
    name = "llvm_toolchain",
    llvm_version = "15.0.6",
)

load("@llvm_toolchain//:toolchains.bzl", "llvm_register_toolchains")

llvm_register_toolchains()

and added to .bazelrc:

# llvm issue https://github.com/bazelbuild/bazel/issues/7260
build --incompatible_enable_cc_toolchain_resolution

Also, just to be sure:

• Does the target you're building work without bazel-toolchain but with --incompatible_enable_cc_toolchain_resolution?

I'm a bit confused here, without the flag --incompatible_enable_cc_toolchain_resolution it compiles, but checking with subcommands I can see it's using the system gcc. I tried also the debugging flag but the output looked a bit ambiguous.

: # @boringssl//:crypto [action 'Compiling src/crypto/x509v3/v3_bitst.c', configuration: 11e15a16ab2ed58d1bea9548e2f8fe647313a513442011e12817a87c774f5f56, execution platform: @local_config_platform//:host]
(cd /home/lromor/.cache/bazel/_bazel_lromor/8a6505bc39ef53295ab5ad911f458639/execroot/ai_ellogon_eslide && \
  exec env - \
    PATH=/home/lromor/.cache/bazelisk/downloads/bazelbuild/bazel-5.4.0-linux-x86_64/bin:/home/lromor/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/usr/local/go/bin \
    PWD=/proc/self/cwd \
  /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -MD -MF bazel-out/k8-fastbuild/bin/external/boringssl/_objs/crypto/v3_bitst.pic.d '-frandom-seed=bazel-out/k8-fastbuild/bin/external/boringssl/_objs/crypto/v3_bitst.pic.o' -fPIC -iquote external/boringssl -iquote bazel-out/k8-fastbuild/bin/external/boringssl -isystem external/boringssl/src/include -isystem bazel-out/k8-fastbuild/bin/external/boringssl/src/include -Wa,--noexecstack '-D_XOPEN_SOURCE=700' -Wall -Werror '-Wformat=2' -Wsign-compare -Wmissing-field-initializers -Wwrite-strings -Wshadow -fno-common '-std=c11' -Wmissing-prototypes -Wold-style-definition -Wstrict-prototypes -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c external/boringssl/src/crypto/x509v3/v3_bitst.c -o bazel-out/k8-fastbuild/bin/external/boringssl/_objs/crypto/v3_bitst.pic.o)
# Configuration: 11e15a16ab2ed58d1bea9548e2f8fe647313a513442011e12817a87c774f5f56
# Execution platform: @local_config_platform//:host

with the flag I get:

	File "/home/lromor/.cache/bazel/_bazel_lromor/8a6505bc39ef53295ab5ad911f458639/external/rules_cc/cc/find_cc_toolchain.bzl", line 68, column 13, in find_cc_toolchain
		fail("In order to use find_cc_toolchain, your rule has to depend on C++ toolchain. See find_cc_toolchain.bzl docs for details.")
Error in fail: In order to use find_cc_toolchain, your rule has to depend on C++ toolchain. See find_cc_toolchain.bzl docs for details.

• Do the targets being built come from rules_cc rules or custom rules?

Standard rules_cc.

---

Regarding the nixos proposal, thank you for answering in such a detailed manner. I understand the issues. I'll wait for the repo owners to make a decision on what would be in their opinion the way to go if any.

[lromor]

Anyways, give me some time for the first issue. I'll try to reproduce it on a minimial project and get back to you!

[lromor]

@rrbutani I made a separate issue the first part!

[siddharthab]

Is this still being worked on?

[lromor]

No, didn't have time for it. But, for posterity, I figured out by passing the variables NIX_LD and NIX_LD_LIBRARY_PATH via

test          --test_env=NIX_LD=$NIX_LD        --test_env=NIX_LD_LIBRARY_PATH=$NIX_LD_LIBRARY_PATH
common      --action_env=NIX_LD=$NIX_LD      --action_env=NIX_LD_LIBRARY_PATH=$NIX_LD_LIBRARY_PATH
common --host_action_env=NIX_LD=$NIX_LD --host_action_env=NIX_LD_LIBRARY_PATH=$NIX_LD_LIBRARY_PATH

works well for most bazel-pulled precompiled stuff. In this way there's no need for changes in the toolchain. What do you think?

[siddharthab]

Thanks for getting back on this. Yes, I think that should be good enough. nix is ultimately just setting up an environment through environment variables. Technically, you can also run bazel under nix, or there might be better integration setups under https://nix-bazel.build/.