Training on Win10, RuntimeError: Could not find MSVC/GCC/CLANG

[adrienchaton]

Hello,

I am trying to train Style GAN 3 on a windows 10 computer with TITAN RTX 24GB:
Driver Version: 496.76 CUDA Version: 11.5

Dependencies seem to be correctly installed (including Ninja), with Pytorch 1.9.0 with CUDA support using their wheel for past release, available for CUDA 11.1 (cf. https://pytorch.org/get-started/previous-versions/).

Regular Pytorch operations run without issues on GPU but there seem to be issues when setting up the custom CUDA operations.

Constructing networks...
Setting up PyTorch plugin "bias_act_plugin"... Failed!
Traceback (most recent call last):
  File "D:\side_projects\microscopic_pix2pix\stylegan3-main\train.py", line 286, in <module>
    main() # pylint: disable=no-value-for-parameter
  File "D:\coding\virtual_envs\venv_MM_SG3\lib\site-packages\click\core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "D:\coding\virtual_envs\venv_MM_SG3\lib\site-packages\click\core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "D:\coding\virtual_envs\venv_MM_SG3\lib\site-packages\click\core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "D:\coding\virtual_envs\venv_MM_SG3\lib\site-packages\click\core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "D:\side_projects\microscopic_pix2pix\stylegan3-main\train.py", line 281, in main
    launch_training(c=c, desc=desc, outdir=opts.outdir, dry_run=opts.dry_run)
  File "D:\side_projects\microscopic_pix2pix\stylegan3-main\train.py", line 96, in launch_training
    subprocess_fn(rank=0, c=c, temp_dir=temp_dir)
  File "D:\side_projects\microscopic_pix2pix\stylegan3-main\train.py", line 47, in subprocess_fn
    training_loop.training_loop(rank=rank, **c)
  File "D:\side_projects\microscopic_pix2pix\stylegan3-main\training\training_loop.py", line 168, in training_loop
    img = misc.print_module_summary(G, [z, c])
  File "D:\side_projects\microscopic_pix2pix\stylegan3-main\torch_utils\misc.py", line 216, in print_module_summary
    outputs = module(*inputs)
  File "D:\coding\virtual_envs\venv_MM_SG3\lib\site-packages\torch\nn\modules\module.py", line 1071, in _call_impl
    result = forward_call(*input, **kwargs)
  File "D:\side_projects\microscopic_pix2pix\stylegan3-main\training\networks_stylegan3.py", line 511, in forward
    ws = self.mapping(z, c, truncation_psi=truncation_psi, truncation_cutoff=truncation_cutoff, update_emas=update_emas)
  File "D:\coding\virtual_envs\venv_MM_SG3\lib\site-packages\torch\nn\modules\module.py", line 1071, in _call_impl
    result = forward_call(*input, **kwargs)
  File "D:\side_projects\microscopic_pix2pix\stylegan3-main\training\networks_stylegan3.py", line 151, in forward
    x = getattr(self, f'fc{idx}')(x)
  File "D:\coding\virtual_envs\venv_MM_SG3\lib\site-packages\torch\nn\modules\module.py", line 1071, in _call_impl
    result = forward_call(*input, **kwargs)
  File "D:\side_projects\microscopic_pix2pix\stylegan3-main\training\networks_stylegan3.py", line 100, in forward
    x = bias_act.bias_act(x, b, act=self.activation)
  File "D:\side_projects\microscopic_pix2pix\stylegan3-main\torch_utils\ops\bias_act.py", line 84, in bias_act
    if impl == 'cuda' and x.device.type == 'cuda' and _init():
  File "D:\side_projects\microscopic_pix2pix\stylegan3-main\torch_utils\ops\bias_act.py", line 46, in _init
    extra_cuda_cflags=['--use_fast_math'],
  File "D:\side_projects\microscopic_pix2pix\stylegan3-main\torch_utils\custom_ops.py", line 84, in get_plugin
    raise RuntimeError(f'Could not find MSVC/GCC/CLANG installation on this computer. Check _find_compiler_bindir() in "{__file__}".')
RuntimeError: Could not find MSVC/GCC/CLANG installation on this computer. Check _find_compiler_bindir() in "D:\side_projects\microscopic_pix2pix\stylegan3-main\torch_utils\custom_ops.py".

Would anyone know how to fix this please ?

I am launching from terminal (Cmder emulator) inside a virtual environment using a command such as:

python D:\side_projects\microscopic_pix2pix\stylegan3-main\train.py --outdir=D:\side_projects\microscopic_pix2pix\stylegan3-main\training_runs\SGAN00_T_512px_bs32 --data=D:\side_projects\MM_Dataset\210106_Blue_Sample_Stitch_Data_SGAN512 --cfg=stylegan3-t --gpus=1 --batch=32 --gamma=8 --batch-gpu=8 --snap=10

Thanks !

[adrienchaton]

I didnt have the c++ development tools installed as part of Visual Studio, which I fixed now, however I keep having an error on the setting up of bias_act_plugin. Are there some further installation fixes I need to take please ?

This error also happens when running the codes from Visual Studio PowerShell, with the same virtual environment.

Constructing networks...
Setting up PyTorch plugin "bias_act_plugin"... Failed!
Traceback (most recent call last):
  File "D:\coding\virtual_envs\venv_MM_SG3\lib\site-packages\torch\utils\cpp_extension.py", line 1672, in _run_ninja_build
    env=env)
  File "C:\Users\adrio\AppData\Local\Programs\Python\Python37\lib\subprocess.py", line 512, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "D:\side_projects\microscopic_pix2pix\stylegan3-main\train.py", line 286, in <module>
    main() # pylint: disable=no-value-for-parameter
  File "D:\coding\virtual_envs\venv_MM_SG3\lib\site-packages\click\core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "D:\coding\virtual_envs\venv_MM_SG3\lib\site-packages\click\core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "D:\coding\virtual_envs\venv_MM_SG3\lib\site-packages\click\core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "D:\coding\virtual_envs\venv_MM_SG3\lib\site-packages\click\core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "D:\side_projects\microscopic_pix2pix\stylegan3-main\train.py", line 281, in main
    launch_training(c=c, desc=desc, outdir=opts.outdir, dry_run=opts.dry_run)
  File "D:\side_projects\microscopic_pix2pix\stylegan3-main\train.py", line 96, in launch_training
    subprocess_fn(rank=0, c=c, temp_dir=temp_dir)
  File "D:\side_projects\microscopic_pix2pix\stylegan3-main\train.py", line 47, in subprocess_fn
    training_loop.training_loop(rank=rank, **c)
  File "D:\side_projects\microscopic_pix2pix\stylegan3-main\training\training_loop.py", line 168, in training_loop
    img = misc.print_module_summary(G, [z, c])
  File "D:\side_projects\microscopic_pix2pix\stylegan3-main\torch_utils\misc.py", line 216, in print_module_summary
    outputs = module(*inputs)
  File "D:\coding\virtual_envs\venv_MM_SG3\lib\site-packages\torch\nn\modules\module.py", line 1071, in _call_impl
    result = forward_call(*input, **kwargs)
  File "D:\side_projects\microscopic_pix2pix\stylegan3-main\training\networks_stylegan3.py", line 511, in forward
    ws = self.mapping(z, c, truncation_psi=truncation_psi, truncation_cutoff=truncation_cutoff, update_emas=update_emas)
  File "D:\coding\virtual_envs\venv_MM_SG3\lib\site-packages\torch\nn\modules\module.py", line 1071, in _call_impl
    result = forward_call(*input, **kwargs)
  File "D:\side_projects\microscopic_pix2pix\stylegan3-main\training\networks_stylegan3.py", line 151, in forward
    x = getattr(self, f'fc{idx}')(x)
  File "D:\coding\virtual_envs\venv_MM_SG3\lib\site-packages\torch\nn\modules\module.py", line 1071, in _call_impl
    result = forward_call(*input, **kwargs)
  File "D:\side_projects\microscopic_pix2pix\stylegan3-main\training\networks_stylegan3.py", line 100, in forward
    x = bias_act.bias_act(x, b, act=self.activation)
  File "D:\side_projects\microscopic_pix2pix\stylegan3-main\torch_utils\ops\bias_act.py", line 84, in bias_act
    if impl == 'cuda' and x.device.type == 'cuda' and _init():
  File "D:\side_projects\microscopic_pix2pix\stylegan3-main\torch_utils\ops\bias_act.py", line 46, in _init
    extra_cuda_cflags=['--use_fast_math'],
  File "D:\side_projects\microscopic_pix2pix\stylegan3-main\torch_utils\custom_ops.py", line 137, in get_plugin
    verbose=verbose_build, sources=cached_sources, **build_kwargs)
  File "D:\coding\virtual_envs\venv_MM_SG3\lib\site-packages\torch\utils\cpp_extension.py", line 1092, in load
    keep_intermediates=keep_intermediates)
  File "D:\coding\virtual_envs\venv_MM_SG3\lib\site-packages\torch\utils\cpp_extension.py", line 1303, in _jit_compile
    is_standalone=is_standalone)
  File "D:\coding\virtual_envs\venv_MM_SG3\lib\site-packages\torch\utils\cpp_extension.py", line 1408, in _write_ninja_file_and_build_library
    error_prefix=f"Error building extension '{name}'")
  File "D:\coding\virtual_envs\venv_MM_SG3\lib\site-packages\torch\utils\cpp_extension.py", line 1682, in _run_ninja_build
    raise RuntimeError(message) from e
RuntimeError: Error building extension 'bias_act_plugin': [1/1] "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\bin\Hostx64\x64/link.exe" bias_act.o bias_act.cuda.o /nologo /DLL c10.lib c10_cuda.lib torch_cpu.lib torch_cuda_cu.lib -INCLUDE:?searchsorted_cuda@native@at@@YA?AVTensor@2@AEBV32@0_N1@Z torch_cuda_cpp.lib -INCLUDE:?warp_size@cuda@at@@YAHXZ torch.lib /LIBPATH:D:\coding\virtual_envs\venv_MM_SG3\lib\site-packages\torch\lib torch_python.lib /LIBPATH:D:\coding\virtual_envs\venv_MM_SG3\Scripts\libs "/LIBPATH:C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.5\lib/x64" cudart.lib /out:bias_act_plugin.pyd
FAILED: bias_act_plugin.pyd
"C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.29.30133\bin\Hostx64\x64/link.exe" bias_act.o bias_act.cuda.o /nologo /DLL c10.lib c10_cuda.lib torch_cpu.lib torch_cuda_cu.lib -INCLUDE:?searchsorted_cuda@native@at@@YA?AVTensor@2@AEBV32@0_N1@Z torch_cuda_cpp.lib -INCLUDE:?warp_size@cuda@at@@YAHXZ torch.lib /LIBPATH:D:\coding\virtual_envs\venv_MM_SG3\lib\site-packages\torch\lib torch_python.lib /LIBPATH:D:\coding\virtual_envs\venv_MM_SG3\Scripts\libs "/LIBPATH:C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.5\lib/x64" cudart.lib /out:bias_act_plugin.pyd
LINK : fatal error LNK1104: cannot open file 'python37.lib'
ninja: build stopped: subcommand failed.

[WestAnderson]

Hey also having the same issue.

Setting up PyTorch plugin "bias_act_plugin"... Failed!

bias_act_plugin just fails to start up

(stylegan3) PS C:\Users\Anderson\Desktop\MachineLearning\stylegan3> python gen_images.py --outdir=out --trunc=1 --seeds=2 --network=https://api.ngc.nvidia.com/v2/models/nvidia/research/stylegan3/versions/1/files/stylegan3-r-afhqv2-512x512.pkl Loading networks from "https://api.ngc.nvidia.com/v2/models/nvidia/research/stylegan3/versions/1/files/stylegan3-r-afhqv2-512x512.pkl"... Generating image for seed 2 (0/1) ... Setting up PyTorch plugin "bias_act_plugin"... Failed! Traceback (most recent call last): File "C:\Users\Anderson\Desktop\MachineLearning\stylegan3\gen_images.py", line 143, in <module> generate_images() # pylint: disable=no-value-for-parameter File "C:\Users\Anderson\.conda\envs\stylegan3\lib\site-packages\click\core.py", line 1128, in __call__ return self.main(*args, **kwargs) File "C:\Users\Anderson\.conda\envs\stylegan3\lib\site-packages\click\core.py", line 1053, in main rv = self.invoke(ctx) File "C:\Users\Anderson\.conda\envs\stylegan3\lib\site-packages\click\core.py", line 1395, in invoke return ctx.invoke(self.callback, **ctx.params) File "C:\Users\Anderson\.conda\envs\stylegan3\lib\site-packages\click\core.py", line 754, in invoke return __callback(*args, **kwargs) File "C:\Users\Anderson\Desktop\MachineLearning\stylegan3\gen_images.py", line 135, in generate_images img = G(z, label, truncation_psi=truncation_psi, noise_mode=noise_mode) File "C:\Users\Anderson\.conda\envs\stylegan3\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl return forward_call(*input, **kwargs) File "<string>", line 503, in forward File "C:\Users\Anderson\.conda\envs\stylegan3\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl return forward_call(*input, **kwargs) File "<string>", line 143, in forward File "C:\Users\Anderson\.conda\envs\stylegan3\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl return forward_call(*input, **kwargs) File "<string>", line 92, in forward File "C:\Users\Anderson\Desktop\MachineLearning\stylegan3\torch_utils\ops\bias_act.py", line 84, in bias_act if impl == 'cuda' and x.device.type == 'cuda' and _init(): File "C:\Users\Anderson\Desktop\MachineLearning\stylegan3\torch_utils\ops\bias_act.py", line 41, in _init _plugin = custom_ops.get_plugin( File "C:\Users\Anderson\Desktop\MachineLearning\stylegan3\torch_utils\custom_ops.py", line 84, in get_plugin raise RuntimeError(f'Could not find MSVC/GCC/CLANG installation on this computer. Check _find_compiler_bindir() in "{__file__}".') RuntimeError: Could not find MSVC/GCC/CLANG installation on this computer. Check _find_compiler_bindir() in "C:\Users\Anderson\Desktop\MachineLearning\stylegan3\torch_utils\custom_ops.py". (stylegan3) PS C:\Users\Anderson\Desktop\MachineLearning\stylegan3>

Hope someone could help. Thanks!

[adrienchaton]

Now I have it working.

Two things that help:
_ deleting the cache of the built torch extensions
_ managing the environment with conda
on linux I can have it running just installing with pip
on windows I could not get the configuration to work without conda
just it needed to add the conda-forge channel to the yml config

[jannehellsten]

I recently found that VS2019 community edition doesn't work with StyleGAN3 but VS2022 does. Alas, using VS2022 required some small changes. In case anyone above is struggling, this change might help:

69c7ef0