Hardware acceleration does not work without LD_LIBRARY_PATH hacks

[Pleune]

Bug description

At this time, the WSL2/WSLg D3D12 hardware acceleration does not work out of the box due to library problems, even though the mesa.drivers package does have D3D12 support enabled by default.

I do not understand exactly what is going on. but there are a few ways to set LD_LIBRARY_PATH as a hack to fix it:

• LD_LIBRARY_PATH=/run/opengl-driver/lib works with MESA_D3D12_DEFAULT_ADAPTER_NAME=Nvidia but not Intel (My laptop has two GPUs)
• LD_LIBRARY_PATH=ub:/run/opengl-driver/lib works with both Intel and Nvidia from above, where the ub folder has some basic library files copied from the WSL2 Ubuntu system - see below for contents

In both examples above, /run/opengl-driver/lib can be swapped with /usr/lib/wsl/lib. I'm not sure what the difference is here, other than that /usr/lib/wsl/lib has many more files.

I believe this could be two problems:

1. the mesa.drivers files are not able to dlopen the WSLg files, since it has no way to know where they are.
2. The Intel version of the D3D12 sub-driver-??? seems to need some additional libraries to link to that are not currently available.

To Reproduce

With the following config:

  programs.nix-ld = {
    enable = true;
    # I am using this nix-ld-rs too:
    #   package = nix-ld-rs.packages."${pkgs.system}".nix-ld-rs;
  };
  wsl = {
    enable = true;
    defaultUser = "${username}";
    nativeSystemd = true;
    useWindowsDriver = true;
    startMenuLaunchers = true;
  };

  hardware.opengl = {
    enable = true;
    driSupport = true;

    extraPackages = with pkgs; [
      mesa.drivers
      libvdpau-va-gl
      vaapiVdpau
    ];
  };

Then run any of the following commands:

glxinfo -B
MESA_D3D12_DEFAULT_ADAPTER_NAME=Intel LD_LIBRARY_PATH=ub:/run/opengl-driver/lib glxinfo -B
MESA_D3D12_DEFAULT_ADAPTER_NAME=Nvidia LD_LIBRARY_PATH=/run/opengl-driver/lib glxinfo -B
MESA_D3D12_DEFAULT_ADAPTER_NAME=Intel LD_LIBRARY_PATH=/usr/lib/wsl/lib glxinfo -B
MESA_D3D12_DEFAULT_ADAPTER_NAME=Nvidia LD_LIBRARY_PATH=ub:/usr/lib/wsl/lib glxinfo -B

The ub folder I am using is: ub.tar.gz

Logs

Example of hardware acceleration working:

> LD_LIBRARY_PATH=ub:/usr/lib/wsl/lib glxinfo -B
name of display: :0
WARNING: dzn is not a conformant Vulkan implementation, testing use only.
WARNING: dzn is not a conformant Vulkan implementation, testing use only.
WARNING: Some incorrect rendering might occur because the selected Vulkan device (Microsoft Direct3D12 (NVIDIA RTX A500 Laptop GPU)) doesn't support base Zink requirements: feats.features.logicOp have_EXT_custom_border_color have_EXT_line_rasterization
glx: failed to create drisw screen
failed to load driver: zink
display: :0  screen: 0
direct rendering: Yes
Extended renderer info (GLX_MESA_query_renderer):
    Vendor: Microsoft Corporation (0xffffffff)
    Device: D3D12 (Intel(R) Iris(R) Xe Graphics) (0xffffffff)
    Version: 24.0.3
    Accelerated: yes
    Video memory: 8146MB
    Unified memory: yes
    Preferred profile: core (0x1)
    Max core profile version: 4.1
    Max compat profile version: 4.1
    Max GLES1 profile version: 1.1
    Max GLES[23] profile version: 3.0

< snip >

What happens without the LD_LIBRARY_PATH hack:

> glxinfo -B
name of display: :0
MESA: error: ZINK: failed to choose pdev
glx: failed to create drisw screen
failed to load driver: zink
display: :0  screen: 0
direct rendering: Yes
Extended renderer info (GLX_MESA_query_renderer):
    Vendor: Mesa (0xffffffff)
    Device: llvmpipe (LLVM 17.0.6, 256 bits) (0xffffffff)
    Version: 24.0.3
    Accelerated: no
    Video memory: 7770MB
    Unified memory: yes
    Preferred profile: core (0x1)
    Max core profile version: 4.5
    Max compat profile version: 4.5
    Max GLES1 profile version: 1.1
    Max GLES[23] profile version: 3.2

< snip >

Where the ub folder has the following files:

> ls ub
libbsd.so.0
libedit.so.2
libmd.so.0
libstdc++.so.6
libtinfo.so.6
libz.so.1

WSL version

WSL version: 2.2.2.0
Kernel version: 5.15.150.1-2
WSLg version: 1.0.61
MSRDC version: 1.2.5105
Direct3D version: 1.611.1-81528511
DXCore version: 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp
Windows version: 10.0.22631.3296

[SuperSandro2000]

/usr/lib/wsl/lib is symlinked into /run/opengl-driver/lib

Can you run the same commands with strace to see if it searches the right directories?

[Pleune]

Here is the output of MESA_D3D12_DEFAULT_ADAPTER_NAME=Nvidia strace -o strace.log glxinfo -B, where llvmpipe is used instead of the GPU

strace.log

I can see many references to /run/opengl-driver/lib here, so it must be searching that path. Maybe the initial d3d12 library is being loaded, but then when the code inside that runs, dlopen fails?

[SuperSandro2000]

It is not searching for d3d12 in the right location.

openat(AT_FDCWD, "/etc/drirc", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/home/mitch/.drirc", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/vcxkk3l25hlix62aivl2prfhpczrvymp-expat-2.6.2/lib/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/zph9xw0drmq3rl2ik5slg0n2frw9lw5m-zlib-1.3.1/lib/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/hqvkggqxqslgy2ahp1zswgh7im2kl5im-libxcb-1.16/lib/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/qmy3imvwp6l129avxifnczw4p3fkdz0j-libX11-1.8.7/lib/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/zmpw448943piz8vdav28j8dqfy54xwjc-libxshmfence-1.3.2/lib/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/ss6gh67xv5jw6jh0l7dwmyx9823wvb60-zstd-1.5.5/lib/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/blw10rx1cayp2n2pkmyihpipifzgj2xq-wayland-1.22.0/lib/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/1birlqjpkhsadz9zxa1in8z1apz7l1x4-systemd-minimal-libs-255.4/lib/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/ixi9ca45786dq510fk6h7crmi9259qcd-libdrm-2.4.120/lib/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/ddwyrxif62r8n6xclvskjyy6szdhvj60-glibc-2.39-5/lib/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/c2yb135iv4maadia5f760b3xhbh6jh61-gcc-13.2.0-lib/lib/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/5xylmnrx48vqlcfv1daj63kql91gflx3-mesa-24.0.3-drivers/lib/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/ddwyrxif62r8n6xclvskjyy6szdhvj60-glibc-2.39-5/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/ddwyrxif62r8n6xclvskjyy6szdhvj60-glibc-2.39-5/lib/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/rxganm4ibf31qngal3j3psp20mak37yy-xgcc-13.2.0-libgcc/lib/glibc-hwcaps/x86-64-v3/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/nix/store/rxganm4ibf31qngal3j3psp20mak37yy-xgcc-13.2.0-libgcc/lib/glibc-hwcaps/x86-64-v3/", 0x7ffdb4a032e0, 0) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/rxganm4ibf31qngal3j3psp20mak37yy-xgcc-13.2.0-libgcc/lib/glibc-hwcaps/x86-64-v2/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/nix/store/rxganm4ibf31qngal3j3psp20mak37yy-xgcc-13.2.0-libgcc/lib/glibc-hwcaps/x86-64-v2/", 0x7ffdb4a032e0, 0) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/rxganm4ibf31qngal3j3psp20mak37yy-xgcc-13.2.0-libgcc/lib/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/nix/store/rxganm4ibf31qngal3j3psp20mak37yy-xgcc-13.2.0-libgcc/lib/", {st_mode=S_IFDIR|0555, st_size=4096, ...}, 0) = 0
futex(0x7fe8aa4b4e1c, FUTEX_WAKE_PRIVATE, 2147483647) = 0
openat(AT_FDCWD, "/nix/store/bhfs2c58cvgfmr5lh2j3p5032sy0bg11-mesa-24.0.3/share/drirc.d", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 4
fstat(4, {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0
getdents64(4, 0x186db00 /* 4 entries */, 32768) = 144
getdents64(4, 0x186db00 /* 0 entries */, 32768) = 0
close(4)                                = 0
openat(AT_FDCWD, "/nix/store/bhfs2c58cvgfmr5lh2j3p5032sy0bg11-mesa-24.0.3/share/drirc.d/00-mesa-defaults.conf", O_RDONLY) = 4

[Pleune]

hmm, libd3d12.so does exist in my nix store in from the wsl-lib and opengl-drivers packages:

> find /nix -name libd3d12.so
/nix/store/4mpn0189n0wrbi6xscricgsrdkx26bmx-wsl-lib/lib/libd3d12.so
/nix/store/gbfz16cazfd57f45654x916g06cwifxf-opengl-drivers/lib/libd3d12.so
/nix/store/1h8c6cx18mwn0i0kcz25mzbbf87mf8jp-opengl-drivers/lib/libd3d12.so
/nix/store/lk3pnpksmdw28m0nlj069vvg63l7ljjk-opengl-drivers/lib/libd3d12.so
/nix/store/0lxfb1j4yi3r0kaggq0hg0ad6q0i57ix-opengl-drivers/lib/libd3d12.so
/nix/store/57nb4gx6pmn46mvc8hlr26jmgm5j6i18-opengl-drivers/lib/libd3d12.so
/nix/store/6xq076h9n17fhb1w3f1pv1flspx1386m-opengl-drivers/lib/libd3d12.so
/nix/store/izbp5v8nvzs18gipcmd31iyywaqx10da-opengl-drivers/lib/libd3d12.so
/nix/store/v4wwzpgiazb98xjb0v4z0qpn4g2jwv84-opengl-drivers/lib/libd3d12.so
/nix/store/xzxjmcny8xdqv7b2sm0ws77zz65x9ylr-opengl-drivers/lib/libd3d12.so
/nix/store/10qydkiksz0zb9xn6fvlwr2s7swanzn1-opengl-drivers/lib/libd3d12.so
/nix/store/qq421zmhp9h5702mqlhirpxbf2jp5xb8-opengl-drivers/lib/libd3d12.so
/nix/store/hsd21kw4lpaxpivdyjrgz8c5vqm5iaq7-opengl-drivers/lib/libd3d12.so

I'm a good bit out of my depth here, but is there a way to add these packages as runtime dependencies for the mesa package?

Setting any of these directories as the LD_LIBRARY_PATH does allow at least the Nvidia d3d12 adapter to work

[Pleune]

Thank you, you have given my enough information to discover the hardware.opengl.setLdLibraryPath = true; option, which makes the Nvidia driver work. Perhaps it would be best to set this as default.

For MESA_D3D12_DEFAULT_ADAPTER_NAME=Intel, I need to also get libedit and libtinfo into /run/opengl-drivers somehow. Perhaps this is a NixOS bug and not a NixOS on WSL bug.

[Pleune]

Alright, for a minimal configuration.nix only snippet, I have the following:

  hardware.opengl = {
    enable = true;
    driSupport = true;
    setLdLibraryPath = true;

    extraPackages = with pkgs; [
      mesa.drivers
      libvdpau-va-gl
      (libedit.overrideAttrs (attrs: {postInstall = (attrs.postInstall or "") + ''ln -s $out/lib/libedit.so $out/lib/libedit.so.2'';}))
    ];
  };

The libedit package only proides libedit.so.5 and libedit.so.0. d3d12 is looking for .2, so for the time being, this will work around that. It seems to pull in all of the other required library files as well.

[sjhaleprogrammer]

Traceback (most recent call last):
  File "/home/samuel/git/stable-diffusion-webui/launch.py", line 48, in <module>
    main()
  File "/home/samuel/git/stable-diffusion-webui/launch.py", line 39, in main
    prepare_environment()
  File "/home/samuel/git/stable-diffusion-webui/modules/launch_utils.py", line 386, in prepare_environment
    raise RuntimeError(
RuntimeError: Torch is not able to use GPU; add --skip-torch-cuda-test to COMMANDLINE_ARGS variable to disable this check
(venv)

how do i get my nvidia gpu to work