Support zig_toolchain building zig from source

[cerisier]

Building zig from source is surprisingly easy, especially in Bazel which translates to:

• 1 cc_library with a bunch @llvm-project targets
• 3 zig_library (--dep std, --dep compiler_mod and --dep aro), all of those **/*.zig
• 1 zig_binary

This opens the door to building zig with rules_zig but more importantly having toolchains that can use a zig built from source from a second toolchain using a prebuilt zig.

This is what we do in https://github.com/hermeticbuild/hermetic-llvm and I was able to very simply adapt this to rules_zig for ZML.

This is game changer for us because we have a lot of non upstreamable commits on top of Zig to build for custom chips that are unknown to everyone.

(cd /private/var/tmp/_bazel_corentinkerisit/cb64bf3e503acdf091b594fa5cf200fb/execroot/_main && \
  exec env - \
  bazel-out/darwin_arm64-opt-exec-ST-e1f9e0a43d02/bin/external/+non_module_deps+zig/zig build-exe --zig-lib-dir external/+non_module_deps+zig/lib --cache-dir /var/tmp/zig-cache --global-cache-dir /var/tmp/zig-cache -O Debug -fno-single-threaded -fllvm -target aarch64-linux-gnu --dep 'bazel_builtin=bazel_builtin_A_S_S_Cadd' '-Madd=add.zig' '-Mbazel_builtin_A_S_S_Cadd=bazel-out/k8-dbg/bin/bazel_builtin_A_S_S_Cadd.zig' '-femit-bin=bazel-out/k8-dbg/bin/add')

---

I would like to discuss introducing this capability to rules_zig.

For the moment, my opinion has been to remove non hermetic local zig installation handling entirely in favor "building zig from source" since they allow the same customization (using a custom zig).

The rawest API is to remove zig_exe_path, zig_lib_path in favor of mandatory cfg="exec" zig_exe, zig_h and zig_lib attributes.

Additionally, I introduce a private bootstrap build_setting to break the dependency loop between prebuilt and from source toolchains.
The default zig_toolchain has a target_settings attribute for built_from_source=False.

When can then let users define a zig_toolchain with a target_settings attribute for built_from_source=True and a rule transition that resets built_from_source=False for all attributes so that the prebuilt toolchain is used to compile whatever label is passed.

Like #643 I think we could add this in multiple steps, 1 raw version to allow advanced usage and later more integrated version with maybe embedding the build files for Zig and the built_from_source toolchain declarations.

WDYT ?

[aherrmann]

That would be a really cool feature to have, definitely! Doing this in multiple steps sounds good, I'm all in favor of small, incremental steps.

The rawest API is to remove zig_exe_path, zig_lib_path in favor of mandatory cfg="exec" zig_exe, zig_h and zig_lib attributes.

Why is it necessary though to remove support for host toolchains? Yes, they are problematic wrt hermeticity. But, removing them, removes some legitimate use-cases, e.g. Nix integration via Nix store paths, or host toolchain in pinned Docker build images.

[cerisier]

I agree in an ideal world we wouldn't remove them but handling both became so complex that I kind of gave up. If you want, I can submit a PR with the current implementation and we can see how we can support the local installation back ?

[aherrmann]

I agree in an ideal world we wouldn't remove them but handling both became so complex that I kind of gave up. If you want, I can submit a PR with the current implementation and we can see how we can support the local installation back ?

Yes, that sounds good.