Use conda compilers on Windows

[dagewa]

It is not clear to me how the conda compiler metapackages work on Windows. When I moved my install of Visual Studio out of the way and tried to bootstrap with this change then the build failed. However I have since realised that I would probably have to uninstall Visual Studio completely to see if this can build from zero, because of the Windows Registry.

@graeme-winter, I would be quite interested to see what a python bootstrap.py does on your Windows machine on this branch, without attempting any install of Visual Studio first.

[dagewa]

...in case it isn't clear, python bootstrap.py with this change without breaking my Visual Studio install does work.

[codecov]

Codecov Report

Merging #1626 (fb85136) into main (2990351) will decrease coverage by 0.00%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main    #1626      +/-   ##
==========================================
- Coverage   66.59%   66.59%   -0.01%     
==========================================
  Files         614      614              
  Lines       68814    68814              
  Branches     9578     9578              
==========================================
- Hits        45827    45825       -2     
- Misses      21056    21057       +1     
- Partials     1931     1932       +1     

[dagewa]

This is kind of here for discussion. Conda compilers were commented out ever since 89de77c, but is there more to that other than we didn't try enabling them at the time?

[ndevenish]

I don't recall us ever really getting conda compilation working on windows, it was probably an experimental thing. Since there's not expectation of it currently working I think there's no problem with this just going in?

[Anthchirp]

No, we did have it working on dxtbx/Azure, but then the caching bug happened and that build is still disabled

[dagewa]

NB I did the test of a fresh python bootstrap.py in a Windows OS that had never seen Visual Studio, and it fails. This is perhaps not surprising. I suppose installing conda compilers is no replacement for agreeing to the EULA for Visual Studio.

(via conda environment) bin\libtbx.scons
Running cmd.exe /C indirection.cmd bin\libtbx.scons.bat -j 8 (in build)

C:\Users\David\sw\cctbx\build>call C:\Users\David\sw\cctbx\conda_base\condabin\activate.bat

C:\Users\David\sw\cctbx\build>conda.bat activate

C:\Users\David\sw\cctbx\build>SET DISTUTILS_USE_SDK=1

C:\Users\David\sw\cctbx\build>SET MSSdk=1

C:\Users\David\sw\cctbx\build>SET "VS_VERSION=15.0"

C:\Users\David\sw\cctbx\build>SET "VS_MAJOR=15"

C:\Users\David\sw\cctbx\build>SET "VS_YEAR=2017"

C:\Users\David\sw\cctbx\build>set "MSYS2_ARG_CONV_EXCL=/AI;/AL;/OUT;/out"

C:\Users\David\sw\cctbx\build>set "MSYS2_ENV_CONV_EXCL=CL"

C:\Users\David\sw\cctbx\build>set "PY_VCRUNTIME_REDIST=\bin\vcruntime140.dll"

C:\Users\David\sw\cctbx\build>set "CXX=cl.exe"

C:\Users\David\sw\cctbx\build>set "CC=cl.exe"

C:\Users\David\sw\cctbx\build>set "VSINSTALLDIR="

C:\Users\David\sw\cctbx\build>for /F "usebackq tokens=*" %i in (`vswhere.exe -nologo -products * -version [15.0,16.0) -property installationPath`) do (set "VSINSTALLDIR=%i\" )

C:\Users\David\sw\cctbx\build>if not exist "" (for /F "usebackq tokens=*" %i in (`vswhere.exe -nologo -products * -requires Microsoft.VisualStudio.Component.VC.v141.x86.x64 -property installationPath`) do (set "VSINSTALLDIR=%i\" ) )

C:\Users\David\sw\cctbx\build>if not exist "" (set "VSINSTALLDIR=C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\" )

C:\Users\David\sw\cctbx\build>if not exist "C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\" (set "VSINSTALLDIR=C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\" )

C:\Users\David\sw\cctbx\build>if not exist "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\" (set "VSINSTALLDIR=C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\" )

C:\Users\David\sw\cctbx\build>if not exist "C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\" (set "VSINSTALLDIR=C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\" )

C:\Users\David\sw\cctbx\build>IF NOT "" == "" (
set "INCLUDE=;"
 set "LIB=;"
 set "CMAKE_PREFIX_PATH=;"
)

C:\Users\David\sw\cctbx\build>call :GetWin10SdkDir

C:\Users\David\sw\cctbx\build>call :GetWin10SdkDirHelper HKLM\SOFTWARE\Wow6432Node  1>nul 2>&1

C:\Users\David\sw\cctbx\build>if errorlevel 1 call :GetWin10SdkDirHelper HKCU\SOFTWARE\Wow6432Node  1>nul 2>&1

C:\Users\David\sw\cctbx\build>if errorlevel 1 call :GetWin10SdkDirHelper HKLM\SOFTWARE  1>nul 2>&1

C:\Users\David\sw\cctbx\build>if errorlevel 1 call :GetWin10SdkDirHelper HKCU\SOFTWARE  1>nul 2>&1

C:\Users\David\sw\cctbx\build>if errorlevel 1 exit /B 1

C:\Users\David\sw\cctbx\build>exit /B 0

C:\Users\David\sw\cctbx\build>for /F %i in ('dir /ON /B "\include\10.*"') DO (SET WindowsSDKVer=%~i )
The system cannot find the file specified.

C:\Users\David\sw\cctbx\build>if errorlevel 1 (echo "Didn't find any windows 10 SDK. I'm not sure if things will work, but let's try..." )  else (echo Windows SDK version found as: "" )
Windows SDK version found as: ""

C:\Users\David\sw\cctbx\build>IF "win-64" == "win-64" (
set "CMAKE_GEN=Visual Studio 15 2017 Win64"
 set "BITS=64"
)  else (
set "CMAKE_GEN=Visual Studio 15 2017"
 set "BITS=32"
)

C:\Users\David\sw\cctbx\build>pushd C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\
The system cannot find the path specified.

C:\Users\David\sw\cctbx\build>CALL "VC\Auxiliary\Build\vcvars64.bat" -vcvars_ver=14.16
The system cannot find the path specified.

C:\Users\David\sw\cctbx\build>popd

C:\Users\David\sw\cctbx\build>IF "" == "" SET "CMAKE_GENERATOR=Visual Studio 15 2017 Win64"

C:\Users\David\sw\cctbx\build>call :GetWin10SdkDirHelper HKLM\SOFTWARE\Wow6432Node  1>nul 2>&1

C:\Users\David\sw\cctbx\build>if errorlevel 1 call :GetWin10SdkDirHelper HKCU\SOFTWARE\Wow6432Node  1>nul 2>&1

C:\Users\David\sw\cctbx\build>if errorlevel 1 call :GetWin10SdkDirHelper HKLM\SOFTWARE  1>nul 2>&1

C:\Users\David\sw\cctbx\build>if errorlevel 1 call :GetWin10SdkDirHelper HKCU\SOFTWARE  1>nul 2>&1

C:\Users\David\sw\cctbx\build>if errorlevel 1 exit /B 1

C:\Users\David\sw\cctbx\build>exit /B 0

C:\Users\David\sw\cctbx\build>shift

C:\Users\David\sw\cctbx\build>bin\libtbx.scons.bat -j 8
scons: Reading SConscript files ...

scons: warning: No version of Visual Studio compiler found - C/C++ compilers most likely not set correctly
File "C:\Users\David\sw\cctbx\build\SConstruct", line 3, in <module>
RuntimeError: cl.exe not on PATH:
  File "C:\Users\David\sw\cctbx\build\SConstruct", line 6:
    SConscript("libtbx/SConscript")
  File "C:\Users\David\sw\cctbx\conda_base\Lib\site-packages\SCons\Script\SConscript.py", line 654:
    return method(*args, **kw)
  File "C:\Users\David\sw\cctbx\conda_base\Lib\site-packages\SCons\Script\SConscript.py", line 591:
    return _SConscript(self.fs, *files, **subst_kw)
  File "C:\Users\David\sw\cctbx\conda_base\Lib\site-packages\SCons\Script\SConscript.py", line 280:
    exec(compile(scriptdata, scriptname, 'exec'), call_stack[-1].globals)
  File "C:\Users\David\sw\cctbx\modules\cctbx_project\libtbx\SConscript", line 307:
    raise RuntimeError("cl.exe not on PATH")
usr+sys time: 0.88 seconds
wall clock time: 1.22 seconds
Process failed with return code 2

C:\Users\David\sw\cctbx>

[Anthchirp]

Not too much of a surprise then. I presumably left the compiler packages out then because they were not required. I would suggest then to revert this PR out again.

Also for reference because it seems that this was forgotten some time ago: we already have a Windows build in https://github.com/dials/dials/blob/main/.azure-pipelines/windows-build.yml
It's currently not enabled because it is slow as it is held back by cbflib not being in conda-forge.

[dagewa]

With the compiler packages installed I get version number 19.16.27045 reported for MS C/C++ compilers, without them I get version 19.28.29336. I haven't seen any difference in the end results yet though - either way fails at runtime in dxtbx NeXus code.