windows build failure

[ryanofsky]

Prerequisite for adding windows support in #53 will be fixing windows build issues. Currently make -C depends HOST=x86_64-w64-mingw32 NO_QT=1 V=1 MULTIPROCESS=1 fails with:

bitcoin/depends/work/build/x86_64-w64-mingw32/libmultiprocess/015e95f7ebaa47619a213a19801e7fffafc56864-9af0c516d6d/src/mp/util.cpp:13:10: fatal error: sys/resource.h: No such file or directory
   13 | #include <sys/resource.h>
      |          ^~~~~~~~~~~~~~~~
compilation terminated.

[ryanofsky]

This comment is just my notes about how to make make -C depends HOST=x86_64-w64-mingw32 work in nix-shell. A complete shell.nix file that works is https://gist.github.com/ryanofsky/d0a9ff32adb5b58f48a2ba771a2a9d74.

I use nixos for development, and the main thing that's been keeping me from implementing IPC support for windows has been build system issues. I didn't want to deal with inconvenience of implementing windows support in a non-nix container, and I never got a cross compiled mingw depends build working in nix-shell until now.

I needed the following changes to make this work:

• Needed to add mingw cross compiler pkgsCross.mingwW64.stdenv.cc package to depsBuildBuild build time dependency list. This makes cross compiler binaries like x86_64-w64-mingw32-g++ available to be used by depends system by adding them to the PATH.

• Workaround 1: Unfortunately the standard stdenv.cc package doesn't work properly and needs overrides:

  depsBuildBuild = [
      (pkgs.pkgsCross.mingwW64.stdenv.cc.override ({
      extraBuildCommands = ''
          printf '%s' ' -L${pkgs.pkgsCross.mingwW64.windows.mcfgthreads}/lib' >> $out/nix-support/cc-ldflags
          printf '%s' ' -I${pkgs.pkgsCross.mingwW64.windows.mcfgthreads.dev}/include' >> $out/nix-support/cc-cflags
          printf '%s' ' -L${pkgs.pkgsCross.mingwW64.windows.mingw_w64_pthreads}/lib' >> $out/nix-support/cc-ldflags
          printf '%s' ' -I${pkgs.pkgsCross.mingwW64.windows.mingw_w64_pthreads}/include' >> $out/nix-support/cc-cflags
      '';
      }))
  ];

  Reasons:

  • The windows.mcfgthreads dependency is needed to fix pkgsCross.mingw32.stdenv.cc provides slightly incomplete linking environment: cannot find -lmcfgthread NixOS/nixpkgs#156343 and mingw cross-compiler is broken when invoked outside of nix NixOS/nixpkgs#144126 which happen when you add mingw cross compiler to depsBuildBuild rather than nativeBuildInputs.
  • Adding stdenv.cc package to nativeBuildInputs instead of depsBuildBuild would avoid that problem because instead of just only adding mingw tools to the PATH variable, it would also add them to other variables like NIX_CFLAGS_COMPILE, allowing them to be called without extra options. But unfortunately with the bitcoin depends system we can't do that, because we need a single shell with a working native compiler and cross compiler, and adding mingw variables to the shell environment would break compilation of native packages.
  • Seperately, the windows.mingw_w64_pthreads dependency is need to avoid file not found from #include "pthread.h" in depends zmq package, since pthread support is optional in mingw.

• Workaround 2: Need to unset CC CXX AR ... variables when calling make -C depends

    shellHook = ''
      unset CC CXX AR TAR RANLIB NM STRIP SHA256SUM DOWNLOAD OBJDUMP DSYMUTIL TOUCH
    '';

  Reason: the default pkgs.mkShell implementation automatically sets CC, CXX etc variables to point to the native compiler, but depends system seems to want these variables to either be empty or to point to the cross compiler, So leaving these these variables sets breaks building of all cross-compiled packages. Another possible workaround might have been to switch from pkg.mkShell to pkgs.pkgsCross.mingwW64.mkShell so these variables would point at the cross compiler, but this would make cross-compiler the default compiler, and make it less convenient to do native builds and cross compiled builds side by side. Ideally the depends system would use HOST_ prefixes for cross compiler variables and this would not be necessary.

• Workaround 3: I needed to run export CMAKE_PREFIX_PATH=$PWD/depends/x86_64-w64-mingw32/native:$CMAKE_PREFIX_PATH before make -C depends HOST=x86_64-w64-mingw32 so the depends native_capnp package takes precedence over the nix capnproto package to avoid #error "Version mismatch between generated code and library headers. You must use the same version of the Cap'n Proto compiler and library." . Another way to fix this would be to drop pkgs.capnproto package from buildInputs in shell.nix, but this would make it harder to do normal builds not using the depends system. Ideally the depends system would configure cmake to prefer its own packages over outside packages and avoid this problem.

I also made a number of mistakes along the way to getting this working:

• Initially I didn't know about the pkgs.pkgsCross.mingwW64 definition and resorted to defining cross = (import <nixpkgs> { crossSystem = pkgs.lib.systems.examples.mingwW64; }) and using cross instead. This seemed to work but was less convenient.

• Before using pkgs.pkgsCross.mingwW64.stdenv.cc tried to use pkgs.pkgsCross.mingwW64.gcc which did not work because this is that package is the GCC package intended to run on windows and produce windows binaries, not run on linux and produce windows binaries. I later switched to pkgs.pkgsCross.mingwW64.buildPackage.gcc which did work, before switching to stdenv.cc which seems like a more complete wrapper around buildPackage.gcc.

• Initially I added the windows.mingw_w64_pthreads package to nativeBuildInputs instead of adding it to stdenv.cc override. This broke native builds because it caused mingw pthread flags to be added to NIX_CFLAGS_COMPILE, CMAKE_INCLUDE_PATH, etc variables.

• I ran into problems trying to unset environment variables with env -u on command lines, instead of unset in shellHook. This mistake led to errors that were very hard to debug, because a .direnv file which was adding back the variables I was trying to unset in bash subprocesses, making the unset variables work but not consistently.