fex: add thunk libs

[kuruczgy]

This will need a compiler targeting x86_64, but hopefully that should be relatively easy within nixpkgs.

Resources:

• FEX wiki: https://wiki.fex-emu.com/index.php/Development:Setting_up_Thunks
• FEX ThunkLibs readme: https://github.com/FEX-Emu/FEX/blob/main/ThunkLibs/README.md
• Fedora package spec: https://src.fedoraproject.org/rpms/fex-emu/blob/rawhide/f/fex-emu.spec

Also watch out for this issue with 32-bit thunk libs: FEX-Emu/FEX#4301

[neobrain]

Also watch out for this issue with 32-bit thunk libs: FEX-Emu/FEX#4301

Luckily this turned out to be false observation. Dynamic linking should be compatible with FEX after all, so no special action is needed for 32-bit library forwarding.

[neobrain]

For local development, I set up a shell.nix that I load using nix-shell shell.nix. Perhaps this could be a useful base for the nixpkgs package as well?

{ pkgs ? import <nixpkgs> { } }:

let
  pkgsCross32 = pkgs.pkgsCross.gnu32;
  pkgsCross64 = pkgs.pkgsCross.gnu64;

  devRootFS = pkgs.buildEnv {
    name = "fex-dev-rootfs";
    paths = [
      pkgsCross64.stdenv.cc.libc_dev
      pkgsCross32.stdenv.cc.libc_dev
      pkgsCross64.stdenv.cc.cc
      pkgsCross32.stdenv.cc.cc
    ];
    ignoreCollisions = true;
    pathsToLink = [
      "/include"
      "/lib"
    ];
  };

  toolchain32 = pkgs.writeText "toolchain_nix_x86_32.txt" "
    set(CMAKE_EXE_LINKER_FLAGS_INIT \"-fuse-ld=lld\")
    set(CMAKE_MODULE_LINKER_FLAGS_INIT \"-fuse-ld=lld\")
    set(CMAKE_SHARED_LINKER_FLAGS_INIT \"-fuse-ld=lld\")
    set(CMAKE_SYSTEM_PROCESSOR i686)
    set(CMAKE_C_COMPILER clang)
    set(CMAKE_CXX_COMPILER clang++)
    set(CLANG_FLAGS \"-nodefaultlibs -nostartfiles -lstdc++ -target i686-linux-gnu -msse2 -mfpmath=sse --sysroot=${devRootFS} -idirafter/usr/include\")
    set(CMAKE_C_COMPILER ${pkgsCross32.buildPackages.clang}/bin/i686-unknown-linux-gnu-clang)
    set(CMAKE_CXX_COMPILER ${pkgsCross32.buildPackages.clang}/bin/i686-unknown-linux-gnu-clang++)
    set(CMAKE_C_FLAGS \"\${CMAKE_C_FLAGS} \${CLANG_FLAGS}\")
    set(CMAKE_CXX_FLAGS \"\${CMAKE_CXX_FLAGS} \${CLANG_FLAGS}\")
  ";

  toolchain = pkgs.writeText "toolchain_nix_x86_64.txt" "
    set(CMAKE_EXE_LINKER_FLAGS_INIT \"-fuse-ld=lld\")
    set(CMAKE_MODULE_LINKER_FLAGS_INIT \"-fuse-ld=lld\")
    set(CMAKE_SHARED_LINKER_FLAGS_INIT \"-fuse-ld=lld\")
    set(CMAKE_SYSTEM_PROCESSOR x86_64)
    set(CMAKE_C_COMPILER clang)
    set(CMAKE_CXX_COMPILER clang++)
    set(CLANG_FLAGS \"-nodefaultlibs -nostartfiles -lstdc++ -target x86_64-linux-gnu --sysroot=${devRootFS} -idirafter/usr/include\")
    set(CMAKE_C_COMPILER ${pkgsCross64.buildPackages.clang}/bin/x86_64-unknown-linux-gnu-clang)
    set(CMAKE_CXX_COMPILER ${pkgsCross64.buildPackages.clang}/bin/x86_64-unknown-linux-gnu-clang++)
    set(CMAKE_C_FLAGS \"\${CMAKE_C_FLAGS} \${CLANG_FLAGS}\")
    set(CMAKE_CXX_FLAGS \"\${CMAKE_CXX_FLAGS} \${CLANG_FLAGS}\")
  ";
in
pkgs.mkShell {
  buildInputs = [
    pkgsCross64.buildPackages.clang
    pkgsCross32.buildPackages.clang
  ];

  shellHook = ''
    echo "Set up dev RootFS at ${devRootFS}"
    echo "Host architecture: $(uname -m)"
    echo ""
    echo "x86 gcc: ${pkgsCross64.buildPackages.clang}/bin/clang"
    echo "x86 gcc: ${pkgsCross32.buildPackages.clang}/bin/clang"
    echo "toolchain: ${toolchain}"
    echo "toolchain32: ${toolchain32}"
    echo ""
    echo "Use \$FEX_CMAKE_TOOLCHAINS to configure CMake."
  '';

  FEX_CMAKE_TOOLCHAINS = "-DX86_32_TOOLCHAIN_FILE=${toolchain32} -DX86_64_TOOLCHAIN_FILE=${toolchain} -DX86_DEV_ROOTFS=${devRootFS}";
  ROOTFS = "${devRootFS}";
}

This will set up clang cross compilers and an x86 dev rootfs with the standard library headers as required by FEX. For the actual build configuration, I just run cmake $FEX_CMAKE_TOOLCHAINS -DBUILD_THUNKS=ON.

[neobrain]

Tagging @andre4ik3 in case this is interesting to them. Library forwarding (misleadingly often called "thunks") improves performance in Vulkan and OpenGL applications, enables support for non-Mesa GPU drivers, and is required for compatibility with some specific applications. So it's definitely a useful feature to have upstream support for :)

[neobrain]

Ended up pushing this through to nixpkgs after all (#413255), but I can't proceed without a plan on how to handle the dlopen() call inherently required when forwarding to host libraries. Any advice would be appreciated.

[andre4ik3]

Closed in #413255