chore(bazel): replace hermetic_cc_toolchain (zig cc) with hermetic-llvm

[basvandijk]

Replaces Uber's hermetic_cc_toolchain 4.3.0 (zig cc) with hermetic-llvm — BCR module llvm 0.8.16, a prebuilt clang/lld (LLVM 22.1.8) zero-sysroot cross toolchain — and removes every /tmp/zig-cache reference.

Why

hermetic_cc_toolchain required a zig cache (/tmp/zig-cache) which proved problematic and had to be patched. A newer version fixed the need for the patch but it failed because of another reason on RBE.

Additionally zig cc required four local patches and a pile of plumbing that all disappear with a plain clang/lld toolchain:

Zig workaround	Why it's gone
hermetic_cc_toolchain_strip.patch (out-of-band host strip -S for determinism) + rust.strip_level_select(none)	clang emits deterministic debug info; rules_rust's default opt = -Cstrip=debuginfo keeps symbols, sheds DWARF
hermetic_cc_toolchain_cache_dir.patch (zig cache race under RBE) + all /tmp/zig-cache mounts/mkdirs	clang/lld has no compile cache; runtimes are ordinary sandboxed Bazel actions
hermetic_cc_toolchain_lto.patch (zig's lld fork warned on rustc's -Wl,-O1)	upstream lld accepts -O1 silently
cc_rs.patch (rewrote LLVM triples into zig naming)	standard triples are correct for clang

Toolchain wiring

• hermetic-llvm's extension has no extra_target_settings, so registration goes through wrapper toolchain()s in //bazel gated on the existing //bazel:use_hermetic_cc. --hermetic_cc=false / --config=afl behave exactly as before (verified with --toolchain_resolution_debug: wrappers rejected → falls through to local_config_cc).
• macOS: hermetic for wasm32 (canister) targets only. The zig none_wasm toolchain had no exec constraints, so canister builds on macs relied on it; the macos_*→none_wasm32 wrappers keep that working (wasm needs no Apple SDK). Native macOS targets stay on the system Xcode toolchain: hermetic-llvm's macOS support (trimmed SDK sysroot, resource-dir handling) is not yet mature enough for the full darwin build — the first CI round failed in tikv-jemalloc-sys's configure-based build script (math.h not found).
• glibc floor stays pinned at gnu.2.31 (zig parity) via //:linux-glibc-2.31 (a @platforms//host-parented platform) + build:linux --platforms/--host_platform. The 2.28 default was tried first per plan, but gettid (glibc 2.30) users in rs/ic_os hit the documented contingency.
• Host-library -L leak class (found by CI): cargo build scripts probing host libs via pkg-config inject -L/usr/lib/x86_64-linux-gnu ahead of the hermetic search paths, making -lc resolve to the host glibc while crt objects come from the hermetic one (undefined __libc_csu_init/__libc_csu_fini). zig was immune (it resolved libc internally). Fixed per crate: pcre2-sys and libusb1-sys now build their bundled C sources (more hermetic than before); devicemapper-sys / libcryptsetup-rs-sys / libsystemd-sys link the host library by explicit path via cc_import host-import repos (the existing @libsystemd pattern) with a pkg-config wrapper (bazel/pkg_config_no_link_metadata.sh) that strips -L/-l from probe output while keeping version checks and bindgen include paths; virt-sys / libcryptsetup-rs are probe-only and just get the wrapper.

Missing-unwinder link class (found by CI, round 2)

hermetic-llvm links with --unwindlib=none and stubs out libgcc_s/libunwind with empty archives; the only provider of the _Unwind_* symbols rust std needs is the toolchain's static runtime (cc_toolchain.static_runtime_lib = LLVM's libc++/libc++abi/libunwind archives). rules_rust drops that runtime in two places — zig was immune because zig cc always injected its own unwinder. Both fixed by a patch on the rules_rust fork (bazel/rules_rust_static_runtime.patch; TODO: upstream to dfinity/rules_rust and drop on the next fork bump):

• Every rust_doc_test failed to link (undefined symbol: _Unwind_Resume) — ~34 of the 50 "Bazel Test All" failures and the single arm64-linux failure. rustdoc test links pass include_link_flags = False, which also skipped the static runtime's -lstatic flags; the doc-test writer additionally didn't rewrite the runtime's -Lnative search paths (which live under the toolchain's configuration root) into runfiles-relative form for the test-time link.
• rust_shared_library silently linked no unwinder — the actual cause of all 17 failing system tests (canister_http_*, firewall_*, api_bn_decentralization_*). rules_rust links the dynamic C++ runtime into dylib/cdylib targets, and hermetic-llvm's dynamic runtime consists of versioned files (libunwind.so.1) that -lunwind never matches, so lld silently fell back to the empty stub archive (shared-library links don't error on undefined symbols). At runtime: dlopen of libnss_icos.so failed with undefined symbol: _Unwind_GetIP → GuestOS couldn't resolve the hostos hostname (provided by that NSS plugin) → nft -f rejected the orchestrator-generated ruleset (it whitelists hostos by name: ip6 saddr { hostos }) → the ip6 filter table never appeared, which is exactly what those tests wait for. Diagnosed by SSHing into a keepalive testnet node: reload_nftables.service in a restart loop with Error: Could not resolve hostname. Fix: prefer the static runtime for dylib/cdylib links whenever the toolchain provides one (local/Xcode toolchains provide none and keep their dynamic-runtime behavior).

Fallout fixed along the way

• rs/canister_sandbox's backtrace.c includes <libunwind.h> (zig bundled it): new //bazel:libunwind_include (via bazel/cc_headers_dir.bzl) exposes @llvm-project//libunwind headers to the cc-rs build script. LLVM libunwind is statically linked by the toolchain runtime libs — verified no new runtime .so dependency.
• aws-lc-sys 0.42.0's memcmp compiler probe ignores CFLAGS and can't compile under a zero-sysroot toolchain. Bumped aws-lc-rs 1.17.3 / aws-lc-sys 0.43.0, where the probe is (correctly) skipped for clang — the miscompile it probes for is GCC-specific.
• rustc 1.97's linker_messages lint surfaced benign clang driver chatter on every link; the lint is allowed in bazelrc (linker errors still fail the build).
• libusb-1.0-0-dev dropped from the dev container: the only consumer (vsock_lib's rusb, for HSM attach) now builds the bundled libusb in both build systems — the vendored feature is enabled on the rusb crate.spec (Bazel) and in vsock_lib's Cargo.toml (cargo). Verified: vsock_host links libusb statically with no dynamic libusb NEEDED; cargo check -p vsock_lib passes.
• Fuzzing simplification unlocked by clang: DEFAULT_SANITIZERS dropped the zig-era -Zexternal-clangrt + hardcoded -Clink-arg=…librustc-stable_rt.asan.a pair — rustc's default mechanism links the same bundled ASan runtime itself under clang/lld, without a path that baked in the compilation mode, the rules_rust canonical repo name, the toolchain channel and the arch. Verified: libfuzzer target links exactly one ASan runtime (nm: one defined __asan_init) and runs 100 iterations; AFL target builds green (unaffected by design).
• rules_cc 0.2.18 → 0.2.22, rules_go 0.61.1 → 0.62.0 (matching MVS resolution with llvm@0.8.16 in the graph).

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR migrates the Bazel C/C++ hermetic toolchain from Uber’s hermetic_cc_toolchain (zig cc) to hermetic-llvm (llvm BCR module 0.8.16), updates supporting Bazel/Rust wiring, and removes all zig-cache plumbing.

Changes:

• Replace zig-based hermetic toolchain registration with hermetic-llvm toolchain extension + wrapper toolchain() targets gated by //bazel:use_hermetic_cc.
• Remove /tmp/zig-cache mounts/setup and delete now-unneeded zig-specific patches/config.
• Add a small Bazel rule to expose libunwind headers to cc-rs build scripts and bump aws-lc-* to versions that work under the new toolchain assumptions.

Reviewed changes

Copilot reviewed 21 out of 25 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
third_party/BUILD.libvirt.bazel	Updates rationale/comments for --allow-shlib-undefined under the new hermetic toolchain context.
third_party/BUILD.libsystemd.bazel	Updates rationale/comments for --allow-shlib-undefined under the new hermetic toolchain context.
third_party/BUILD.e2fsprogs.bazel	Simplifies compatibility comment (tools are Linux-only in practice).
rs/canister_sandbox/BUILD.bazel	Adds CFLAGS include path + data dep to supply libunwind.h for cc-rs compilation.
MODULE.bazel	Swaps module deps to llvm@0.8.16, bumps rules_cc/rules_go, and reworks toolchain extension setup/registration.
ci/container/init.sh	Removes creation/permissioning of /tmp/zig-cache.
ci/container/Dockerfile	Updates linker-related comment to reflect clang/lld toolchain; keeps ld.lld symlink.
ci/container/container-run.sh	Removes zig-cache bind mount and directory creation.
Cargo.lock	Bumps aws-lc-rs/aws-lc-sys versions/checksums.
Cargo.Bazel.toml.lock	Mirrors the aws-lc-* version/checksum bumps for Bazel cargo integration.
Cargo.Bazel.json.lock	Updates crate metadata/links for aws-lc-* and removes cc patching entries.
bazel/rust.MODULE.bazel	Removes zig-era strip override and cc crate patch; updates sanitizer comment.
bazel/hermetic_cc_toolchain_strip.patch	Deletes zig-toolchain determinism workaround patch.
bazel/hermetic_cc_toolchain_lto.patch	Deletes zig-lld warning suppression patch.
bazel/hermetic_cc_toolchain_cache_dir.patch	Deletes zig-cache race workaround patch.
bazel/fuzz_testing.bzl	Updates sanitizer comment to reflect clang toolchain context.
bazel/conf/.bazelrc.build	Removes zig-cache-related sandbox/cache config; adds rustc lint allow + macOS archiver flag.
bazel/cc_rs.patch	Deletes zig target-triple rewrite patch for cc-rs.
bazel/cc_headers_dir.bzl	Introduces rule to materialize headers into a directory artifact for build scripts.
bazel/BUILD.bazel	Adds libunwind_include target and wrapper hermetic-llvm toolchains gated on :use_hermetic_cc.
bazel/bazel_clean.sh	Removes cleanup of /tmp/zig-cache.
.github/workflows/ci-main.yml	Removes CI step creating /tmp/zig-cache.
.github/actions/bazel/bin/bazel	Removes zig-cache setup from the GitHub Action wrapper.
.devcontainer/devcontainer.json	Removes zig-cache directory creation and bind mount.
.claude/skills/run-in-dev-container/SKILL.md	Updates documentation to remove zig-cache mention.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[github-actions]

Run URL: https://github.com/dfinity/ic/actions/runs/30750926226

New container images with tag: 827366520f53ae0ab466ed38c5103df6d6726f45aa5789be235bfb5c06797ccb
ic-build: sha256:2a4aed7210bdf3522219410bc14c0ca8161206c02c3cc9f24409f2cb5233f9b5
ic-dev: sha256:484507d3a8cbd75d81f4cc2e990efdde7a780fefacd6d47d0362032f1bfd7968

[Copilot]

Pull request overview

Copilot reviewed 43 out of 47 changed files in this pull request and generated no new comments.

Suppressed comments (2)

bazel/BUILD.bazel:89

• The wrapper toolchains construct OS constraint labels from the string "macos" (via "@platforms//os:" + exec_os/target_os). Elsewhere in the repo, macOS constraints are consistently referenced as @platforms//os:osx (e.g. //:macos-intel in BUILD.bazel), so these toolchains may not match on Darwin and wasm-on-mac toolchain fallback could break.

        exec_compatible_with = [
            "@platforms//os:" + exec_os,
            "@platforms//cpu:" + exec_cpu,
        ],
        target_compatible_with = [
            "@platforms//os:" + target_os,
            "@platforms//cpu:" + target_cpu,
        ],

MODULE.bazel:1387

• The comment "Use libvirt from the host environment" now sits above the cryptsetup/devmapper repositories, which is misleading and makes it harder to understand why these repos are declared. Consider adding a dedicated comment for cryptsetup/devmapper and keeping the libvirt comment directly above the libvirt repo.

# Use libvirt from the host environment
# Used for managing virtual machines programmatically.
new_local_repository(
    name = "cryptsetup",
    build_file = "@//third_party:BUILD.cryptsetup.bazel",
    path = "/usr",
)

[Copilot]

Pull request overview

Copilot reviewed 44 out of 48 changed files in this pull request and generated no new comments.

Suppressed comments (2)

bazel/cc_headers_dir.bzl:24

• The run_shell script doesn't enable set -e/pipefail, so a failed cp/mkdir could be silently ignored and the action would still succeed with an incomplete header directory. Adding set -euo pipefail makes failures propagate and avoids hard-to-debug missing-header errors later in the build.

        command = """
out="$1"; strip="$2"; shift 2
for f in "$@"; do

MODULE.bazel:1382

• This comment block now sits above the cryptsetup/devmapper new_local_repository() declarations, so it no longer accurately describes the code that follows (it reads as if cryptsetup is libvirt). Updating the comment to cover all the host-provided libraries here will avoid confusion when editing this section later.

# Use libvirt from the host environment
# Used for managing virtual machines programmatically.

[Copilot]

Pull request overview

Copilot reviewed 45 out of 49 changed files in this pull request and generated no new comments.

Suppressed comments (1)

bazel/cc_headers_dir.bzl:30

• The run_shell script doesn't enable set -e, so a failed mkdir/cp inside the loop could be ignored and the action might still exit 0 (depending on the last command executed). This could silently produce an incomplete headers directory and lead to confusing downstream compile failures. Consider enabling set -eu at the top of the script so the action fails immediately on errors.

        command = """
out="$1"; strip="$2"; shift 2
for f in "$@"; do
    rel="${f#*"$strip"}"
    [ "$rel" = "$f" ] && continue
    mkdir -p "$out/$(dirname "$rel")"
    cp -L "$f" "$out/$rel"
done
""",

[basvandijk]

Would bazel/rules_rust_static_runtime.patch still be needed with rules_rs?

hermetic-llvm's README recommends using rules_rs for Rust + hermetic-llvm interop, so I looked into whether switching to it would obsolete this PR's rules_rust patch.

TL;DR: yes — with rules_rs the patch would no longer be required. rules_rs isn't an independent ruleset: it's a layer on top of a pinned hermeticbuild/rules_rust fork (currently hermeticbuild/rules_rust@0105631), and that fork already fixes both problem classes the patch addresses — just by different means. One of the two fixes would move from "automatic" to a per-target attribute we'd need to set.

Recap: why the patch exists

hermetic-llvm links with --unwindlib=none and stubs libgcc_s/libunwind as empty archives, so the only provider of the _Unwind_* symbols rust std needs is the cc_toolchain's static_runtime_lib (LLVM's libc++/libc++abi/libunwind archives). Upstream rules_rust 0.71.3 drops that runtime in two places:

1. Doc tests pass include_link_flags = False, skipping the static runtime's -lstatic flags (every rust_doc_test fails with undefined symbol: _Unwind_Resume), and the doc-test script writer doesn't rewrite the runtime's -Lnative paths into runfiles-relative form.
2. dylib/cdylib links use the dynamic runtime via -ldylib=unwind, which never matches hermetic-llvm's versioned libunwind.so.1 — lld silently links no unwinder and the breakage only surfaces at dlopen time (the libnss_icos.so / GuestOS failure).

How the hermeticbuild fork fixes the same things

• Doc tests: their rustdoc.bzl passes include_link_flags = is_test instead of hardcoded False, so the static runtime's link flags flow through construct_arguments naturally. And rust_doc_test was redesigned to run rustdoc as a build action (a RustdocTest action capturing stdout/stderr/exit code) rather than a script that relinks at test runtime — which eliminates the path-stripping problem (this patch's third hunk) by construction. They even carry a regression test, rustdoc_musl_unwind_link_flags_test, asserting the libunwind link arg is present on the doc-test action.
• dylib/cdylib: the fork still defaults to the dynamic runtime, but on non-Windows it links it by full path via -Clink-arg=<path> instead of -ldylib=<name> (their comment: --as-needed could drop libunwind before libstd references it). A full path matches libunwind.so.1 fine, so the "silently missing unwinder" link-time failure mode is gone. It also adds those libs to the rpath.

The one caveat for us

The fork's dylib fix links the runtime dynamically — the resulting .so gets a NEEDED libunwind.so.1 entry, and the rpath points into the Bazel execroot. For shared libraries deployed outside Bazel (exactly libnss_icos.so on GuestOS, where hermetic-llvm's libunwind.so.1 doesn't exist on the image), that would still break at runtime. The fork's answer is an opt-in attribute on rust_shared_library:

cc_runtime_linkage = "static"  # embed libc++/libc++abi/libunwind → self-contained .so

So instead of this patch's "prefer static whenever the toolchain has one" global default, we'd set this on the handful of deployed rust_shared_library targets. Arguably cleaner (explicit, no fork patch), but it's a behavioral difference to remember — forgetting it on a new deployed cdylib reproduces the failure, though now at dlopen with a missing library (loud, obvious) rather than a missing symbol.

Bigger picture

Dropping the patch wouldn't be the hard part of that switch. Moving to rules_rs means replacing the archive_override on dfinity/rules_rust in bazel/rust.MODULE.bazel with hermeticbuild's fork (worth auditing what else the dfinity fork carries), and — the big one — replacing crate_universe with rules_rs's crates.from_cargo, which consumes Cargo.lock directly and would obsolete the Cargo.Bazel.*.lock files and the bazel-pin.sh workflow entirely. rules_rs's rules_rust extension does accept extra patches on the fork, so there's an escape hatch if something else comes up.

Short of that migration, the existing TODO (upstream this patch to the dfinity/rules_rust fork and drop it on the next fork bump) remains the low-effort path.

🤖 Generated with Claude Code