Segmentation fault when patching libcudart.so.10.2.89

[OmnipotentEntity]

Describe the bug

When attempting to build a specific version of CUDA, patchelf segfaults.

patching /nix/store/8zma96haw8rg2nhhkh0dczh0afwnrj96-cudatoolkit-10.2.89/libnvvp/plugins/org.eclipse.equinox.launcher.gtk.linux.x86_64_1.1.200.v20140603-1326/eclipse_1605.so...
patching /nix/store/8zma96haw8rg2nhhkh0dczh0afwnrj96-cudatoolkit-10.2.89/libnvvp/nvvp...
patching /nix/store/pp08fkvf8l60kvcm77wav88pvnkarv81-cudatoolkit-10.2.89-lib/lib/libcudart.so.10.2.89...
/nix/store/f7g5w1gnybmqqaf2ch2ynba1grlayxxf-stdenv-linux/setup: line 100:   523 Segmentation fault      (core dumped) patchelf --set-rpath "$rpath2" --force-rpath $i
builder for '/nix/store/753pmghiqppd12sdj9ql0r381zlvr2ms-cudatoolkit-10.2.89.drv' failed with exit code 139

Steps To Reproduce

Because CUDA takes forever to rebuild, I haven't had an opportunity able to pare down a minimal reproduction. I've uploaded a copy of the offending .so file. And here is the shell.nix file I used to trigger it:

with import "/home/omnipotententity/work/nixpkgs" {};

stdenv.mkDerivation {
  name = "pip-env";
  buildInputs = [
    # System requirements.
    readline
    SDL2
    SDL2_gfx
    SDL2_image
    SDL2_mixer
    SDL2_net
    SDL2_ttf
    xorg.libX11
    xorg.libXext
    xorg.libXrender
    stdenv.cc.cc.lib
    mtdev
    libGL

    # Python requirements (enough to get a virtualenv going).
    python3Full
    python3Packages.virtualenv
    python3Packages.pip

    katagoWithCuda
  ];

  propagatedBuildInputs = [
    python3Packages.setuptools
    xclip
  ];

  src = null;
  shellHook = ''
    # Allow the use of wheels.
    SOURCE_DATE_EPOCH=$(date +%s)

    # Augment the dynamic linker path
    export LD_LIBRARY_PATH=''${LD_LIBRARY_PATH:+$LD_LIBRARY_PATH:}${readline}/lib:/run/opengl-driver/lib:/run/opengl-driver-32/lib:${SDL2}/lib:${SDL2_gfx}/lib:${SDL2_image}/lib:${SDL2_mixer}/lib:${SDL2_net}/lib:${SDL2_ttf}/lib:${xorg.libX11}/lib:${xorg.libXext}/lib:${stdenv.cc.cc.lib}/lib:${mtdev}/lib:${libGL}/lib:${xorg.libXrender}/lib
  '';
}

It should be noted that the import is specifically referring my local copy of the current HEAD of nixpkgs staging. e32bf6f

Expected behavior

Should never segfault.

patchelf --version output

patchelf 0.13.20210805.a949ff2

Additional context

I do not know enough about nixos internals to know exactly what command arguments are being used, though I did make an attempt to figure it out.

[Mic92]

Can you put set -x in preFixup so we can see all the arguments passed to patchelf? In theory we need only the concrete rpath stored in $rpath2 to reproduce the bug given the binary.

[Mic92]

so far this works:

╭─[~/git/patchelf]─[github]
╰─ % patchelf --set-rpath /lib libcudart.so.10.2.89

╭─[~/git/patchelf]─[github]
╰─ % patchelf --print-rpath libcudart.so.10.2.89
/lib

[OmnipotentEntity]

I apologize, things have been hectic for me recently, and I completely forgot I filed this bug. That being said, adding

preFixup = ''
  set -x
'';

Does not seem to give me any additional output.

[OmnipotentEntity]

Here is a "minimal" reproduction. The particular commit is just the commit that I happened to pull today before verifying the problem still exists.

with import
  (builtins.fetchTarball {
    name = "pkgs";
    url = "https://github.com/NixOS/nixpkgs/archive/d370d9ee99c1ae8e7c5e71d2722309eb645fc3df.tar.gz";
    sha256 = "0svi17k9bsv5cqjjxzicdxq2kwa0iqsw4p49qw52f9rxhyap9y21";
  }) {};

pkgs.stdenv.mkDerivation (with pkgs; {
  name = "env";

  preFixup = ''
    set -x
  '';

  buildInputs = [
    cudnn_cudatoolkit_11_2
  ];
})

[Mic92]

The set -x needs to be in cudatoolkit not in your nix-shell.

The derivation tries to set an empty rpath like this:

patchelf --set-rpath "" --force-rpath /nix/store/ra4zy5g6xlvxns98djssz0qadx31bqd8-cudatoolkit-11.2.1-lib/lib/libcudart.so.11.2.146

this is the work around that does not crash:

patchelf --remove-rpath /nix/store/ra4zy5g6xlvxns98djssz0qadx31bqd8-cudatoolkit-11.2.1-lib/lib/libcudart.so.11.2.146

[Mic92]

diff --git a/pkgs/development/compilers/cudatoolkit/common.nix b/pkgs/development/compilers/cudatoolkit/common.nix
index c8a1964814e..b7736821e59 100644
--- a/pkgs/development/compilers/cudatoolkit/common.nix
+++ b/pkgs/development/compilers/cudatoolkit/common.nix
@@ -193,11 +193,10 @@ stdenv.mkDerivation rec {
           --set-interpreter "''$(cat $NIX_CC/nix-support/dynamic-linker)" $i
       fi
       if [[ $i =~ libcudart ]]; then
-        rpath2=
+        patchelf --remove-rpath $i
       else
-        rpath2=$rpath:$lib/lib:$out/jre/lib/amd64/jli:$out/lib:$out/lib64:$out/nvvm/lib:$out/nvvm/lib64
+        patchelf --set-rpath "$rpath:$lib/lib:$out/jre/lib/amd64/jli:$out/lib:$out/lib64:$out/nvvm/lib:$out/nvvm/lib64" --force-rpath $i
       fi
-      patchelf --set-rpath "$rpath2" --force-rpath $i
     done < <(find $out $lib $doc -type f -print0)
   '';

[Mic92]

fixed in #319

[OmnipotentEntity]

Thanks!