Support bazel-central-registry's rules_perl on `nixos`

[shixiong-jiang]

Hello rules_perl owner,

I have a question about rule_perl .
(I did post this question under bazelbuild/bazel-central-registry#3565 too)
We are using bazel to compile cc under nix system.
However, the rule_perl is broken.

You can reproduce it with a nix docker container.

1. Launch a nix docker container on Linux

mkdir /tmp/example; cd /tmp/example

## 1.1. Create a nix config.
## I installed the `pkgs.libcrypt` at first. 
## However, it only has `libcrypt.so.2`, but `rule_perl` built perl wants `libcrypto.so.1`. Only `libxcrypt-legacy` includes it.
## See https://search.nixos.org/packages?channel=unstable&show=libxcrypt-legacy&from=0&size=50&sort=relevance&type=packages&query=libxcrypt

## Here, I also ask nix to install perl by itself. So, we can compare nix install perl vs rule_perl.

cat <<EOF > default.nix
{ pkgs ? import <nixpkgs> {} }:
pkgs.mkShell {
  buildInputs = [
    pkgs.vim
    pkgs.bazel_7
    pkgs.libxcrypt-legacy
    pkgs.perl
  ];
}
EOF

## 1.2. create an example
cat <<EOF > MODULE.bazel
bazel_dep(name = "rules_perl", version = "0.2.5")
bazel_dep(name = "openssl", version = "3.3.1.bcr.1")
EOF

## 1.3. Start a nix docker container

sudo docker run --rm \
  -v /tmp/example:/tmp/example \
  --name my-nix-container -it nixos/nix bash

2. Within the nix, bazel build

cd /tmp/example
nix-channel --update
nix-shell
bazel build --sandbox_debug  @openssl//:all

It fails with this error

/nix/store/fd118hwh7d1ncib4mdw56ylv3g9k0iyj-bash-5.2p37/bin/bash: line 3: external/rules_perl~~perl_repositories~perl_linux_amd64/bin/perl: cannot execute: required file not found

3. Debug

3.1. Run perl directly. It fails too.

## Path is copied from `bazel build --sandbox_debug`'s output 
/root/.cache/bazel/_bazel_root/89a35363ec8de7131a16c2ed7419999a/sandbox/processwrapper-sandbox/48/execroot/_main/external/rules_perl~~perl_repositories~perl_linux_amd64/bin/perl

ldd the perl, libcrypt.so.1 is not found.

ldd /root/.cache/bazel/_bazel_root/89a35363ec8de7131a16c2ed7419999a/sandbox/processwrapper-sandbox/48/execroot/_main/external/rules_perl~~perl_repositories~perl_linux_amd64/bin/perl
	linux-vdso.so.1 (0x00007f508cbdf000)
	libpthread.so.0 => /nix/store/81mi7m3k3wsiz9rrrg636sx21psj20hc-glibc-2.40-66/lib/libpthread.so.0 (0x00007f508cbd4000)
	libdl.so.2 => /nix/store/81mi7m3k3wsiz9rrrg636sx21psj20hc-glibc-2.40-66/lib/libdl.so.2 (0x00007f508cbcf000)
	libm.so.6 => /nix/store/81mi7m3k3wsiz9rrrg636sx21psj20hc-glibc-2.40-66/lib/libm.so.6 (0x00007f508cae6000)
	libcrypt.so.1 => not found
	libutil.so.1 => /nix/store/81mi7m3k3wsiz9rrrg636sx21psj20hc-glibc-2.40-66/lib/libutil.so.1 (0x00007f508cadf000)
	libc.so.6 => /nix/store/81mi7m3k3wsiz9rrrg636sx21psj20hc-glibc-2.40-66/lib/libc.so.6 (0x00007f508c800000)
	/lib64/ld-linux-x86-64.so.2 => /nix/store/81mi7m3k3wsiz9rrrg636sx21psj20hc-glibc-2.40-66/lib64/ld-linux-x86-64.so.2 (0x00007f508cbe1000)

3.2. set LD_LIBRARY_PATH to include libcrypt.so.1

ls /nix/store/*libxcrypt*/lib

Output

/nix/store/d550hx0zzirlbww967k0d0xs8asrb3ia-libxcrypt-4.4.38/lib:
libcrypt.la  libcrypt.so  libcrypt.so.2  libcrypt.so.2.0.0  pkgconfig

/nix/store/flvyd9qpx5gk6pr9l7ki0s841cx717ny-libxcrypt-4.4.36/lib:
libcrypt.la  libcrypt.so  libcrypt.so.2  libcrypt.so.2.0.0  pkgconfig

/nix/store/hcsh4083lvy8sim1ry04r83czfz1r1h6-libxcrypt-4.4.38/lib:
libcrypt.la  libcrypt.so  libcrypt.so.1  libcrypt.so.1.1.0  libxcrypt.so  pkgconfig

Only /nix/store/hcsh4083lvy8sim1ry04r83czfz1r1h6-libxcrypt-4.4.38/lib has libcrypt.so.1. Use it.

(export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/nix/store/hcsh4083lvy8sim1ry04r83czfz1r1h6-libxcrypt-4.4.38/lib; ldd /root/.cache/bazel/_bazel_root/89a35363ec8de7131a16c2ed7419999a/sandbox/processwrapper-sandbox/48/execroot/_main/external/rules_perl~~perl_repositories~perl_linux_amd64/bin/perl)

ldd output looks fine.

	linux-vdso.so.1 (0x00007f5c64300000)
	libpthread.so.0 => /nix/store/81mi7m3k3wsiz9rrrg636sx21psj20hc-glibc-2.40-66/lib/libpthread.so.0 (0x00007f5c642f5000)
	libdl.so.2 => /nix/store/81mi7m3k3wsiz9rrrg636sx21psj20hc-glibc-2.40-66/lib/libdl.so.2 (0x00007f5c642f0000)
	libm.so.6 => /nix/store/81mi7m3k3wsiz9rrrg636sx21psj20hc-glibc-2.40-66/lib/libm.so.6 (0x00007f5c64207000)
	libcrypt.so.1 => /nix/store/hcsh4083lvy8sim1ry04r83czfz1r1h6-libxcrypt-4.4.38/lib/libcrypt.so.1 (0x00007f5c641cb000)
	libutil.so.1 => /nix/store/81mi7m3k3wsiz9rrrg636sx21psj20hc-glibc-2.40-66/lib/libutil.so.1 (0x00007f5c641c4000)
	libc.so.6 => /nix/store/81mi7m3k3wsiz9rrrg636sx21psj20hc-glibc-2.40-66/lib/libc.so.6 (0x00007f5c63e00000)
	/lib64/ld-linux-x86-64.so.2 => /nix/store/81mi7m3k3wsiz9rrrg636sx21psj20hc-glibc-2.40-66/lib64/ld-linux-x86-64.so.2 (0x00007f5c64302000)

3.3. try perl again with LD_LIBRARY_PATH

However, perl still fail

(export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/nix/store/hcsh4083lvy8sim1ry04r83czfz1r1h6-libxcrypt-4.4.38/lib; /root/.cache/bazel/_bazel_root/89a35363ec8de7131a16c2ed7419999a/sandbox/processwrapper-sandbox/48/execroot/_main/external/rules_perl~~perl_repositories~perl_linux_amd64/bin/perl)

Output

bash: /root/.cache/bazel/_bazel_root/89a35363ec8de7131a16c2ed7419999a/sandbox/processwrapper-sandbox/48/execroot/_main/external/rules_perl~~perl_repositories~perl_linux_amd64/bin/perl: cannot execute: required file not found

4. Root cause

readelf -Wl /root/.cache/bazel/_bazel_root/2190073a42434474a8b7f2bcd3c924b6/sandbox/processwrapper-sandbox/48/execroot/_main/external/rules_perl~~perl_repositories~perl_linux_amd64/bin/perl | grep "Requesting program interpreter"

Output

      [Requesting program interpreter: /lib64/ld-linux-x86-64.so.2]

It should be related to the path of the interpreter.
It is supposed to be /nix/store/81mi7m3k3wsiz9rrrg636sx21psj20hc-glibc-2.40-66/lib64/ld-linux-x86-64.so.2.

Explicitly add the /nix/store/81mi7m3k3wsiz9rrrg636sx21psj20hc-glibc-2.40-66/lib64/ld-linux-x86-64.so.2 in front of perl, works

/nix/store/81mi7m3k3wsiz9rrrg636sx21psj20hc-glibc-2.40-66/lib64/ld-linux-x86-64.so.2 /root/.cache/bazel/_bazel_root/2190073a42434474a8b7f2bcd3c924b6/sandbox/processwrapper-sandbox/48/execroot/_main/external/rules_perl~~perl_repositories~perl_linux_amd64/bin/perl -v

Output

This is perl 5, version 36, subversion 0 (v5.36.0) built for x86_64-linux
...

5. Do you have any idea how to fix it?

A tricky workaround

Adding /nix/store/81mi7m3k3wsiz9rrrg636sx21psj20hc-glibc-2.40-66/lib64/ld-linux-x86-64.so.2 in front of bazel build command doesn't help for sure.

patchelf changes the GLIBC path. It fixes the perl, but this option is too trivial.

patchelf --set-interpreter \
  /nix/store/81mi7m3k3wsiz9rrrg636sx21psj20hc-glibc-2.40-66/lib64/ld-linux-x86-64.so.2 \
  /root/.cache/bazel/_bazel_root/2190073a42434474a8b7f2bcd3c924b6/sandbox/processwrapper-sandbox/48/execroot/_main/external/rules_perl~~perl_repositories~perl_linux_amd64/bin/perl

I think the right way to make perl also load the correct GLIBC when compile.
I went through perl's bazel-central-registry code, but I didn't figure out how to fix it.

Thanks a lot

[skeletonkey]

@shixiong-jiang - Thank you for your in-depth description of what we are missing. Perl likes to be installed on the host and that is accelerated when C libraries are brought into the mix.

I'm following you steps but am coming to different outcomes. Please let me know what I should try next:

• ran step 1 - no issue
• ran step 2 - no issue

erik $ sudo docker run --rm \
  -v /tmp/example:/tmp/example \
  --name my-nix-container -it nixos/nix bash
bash-5.2# cd /tmp/example
nix-channel --update
nix-shell
bazel build --sandbox_debug  @openssl//:all
unpacking 1 channels...
these 76 paths will be fetched (571.96 MiB download, 1221.81 MiB unpacked):
  /nix/store/4izi23kn57b3cw47jddzq14wcv01b5y5-acl-2.3.2
 ... <snip>...
copying path '/nix/store/5p64m5v5p8x12gmpxb8dsrvh93ydsz78-bazel-7.4.1' from 'https://cache.nixos.org'...

[nix-shell:/tmp/example]# 

I'm working on

erik $ uname -a
Linux stewart 6.8.0-52-generic #53-Ubuntu SMP PREEMPT_DYNAMIC Sat Jan 11 00:06:25 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Let me know if I can provide any more informaiton,

Erik

[fionera]

Better solution would be to build a fully static perl binary. I did that with Nix, but it should also be possible without

[hzeller]

I ran into the same issue. Yes I also agree a fully statically linked Perl is the only way to make rules_perl somewhat hermetic (and work on nix and other random linux distros).

[fionera]

Just as a small idea on how I did it. I registered a custom toolchain based on the nix built perl.

[shixiong-jiang]

@skeletonkey

It is because: the stdout is too many.

Run last command one more time, you will see the error.

bazel build --sandbox_debug  @openssl//:all

[shixiong-jiang]

@fionera

IIUC, you

1. create your owned tool chain.
2. Then, in the BUILD file, you will set tool chain to the one you defined?

I am trying to build this library. https://registry.bazel.build/modules/openssl
So, I need to write a patch to inject the customized perl tool chain in their BUILD file?

[fionera]

@shixiong-jiang You can register your own toolchain for rules_perl, yes. You don't need to patch anything. You could theoretically register you host env perl binary. In my usecase I actually also had to built openssl, so you can take a look at the linked changes to see what I did