MSBuild can't find the project file for ATEN

[franchenstein]

I am trying to install the latest version of PyTorch for Windows 10 by using cpu.bat, as my computer doesn't have a CUDA-compatible GPU. While running the script using VS prompt, I get the following error when it tries to build ATEN:

-- Build files have been written to: C:/.../pytorch-scripts/pytorch/aten/build

C:\...\pytorch-scripts\pytorch\aten\build>msbuild INSTALL.vcxproj /p:Configuration=Release
Microsoft (R) Build Engine version 15.6.85.37198 for .NET Framework
Copyright (C) Microsoft Corporation. All rights reserved.

MSBUILD : error MSB1009: Project file does not exist.
Switch: INSTALL.vcxproj

C:\...\pytorch-scripts\pytorch\aten\build>IF NOT 1 == 0 exit 1

but I can't find this file anywhere in the directory. Where is it supposed to be?

[peterjc123]

This is not the actual error. It must be occurred before. Could you find it?

[franchenstein]

You're right. I'm sorry, I didn't notice the error before that. I had the following message:

-- Configuring done
CMake Error at src/ATen/cpu/tbb/CMakeLists.txt:239 (add_library):
  Cannot find source file:

    C:/.../pytorch-scripts/pytorch/aten/src/ATen/cpu/tbb/tbb_remote/src/tbb/ia32-masm/itsx.asm src/tbb/ia32-masm/lock_byte.asm

  Tried extensions .c .C .c++ .cc .cpp .cxx .cu .m .M .mm .h .hh .h++ .hm
  .hpp .hxx .in .txx


CMake Error at src/ATen/cpu/tbb/CMakeLists.txt:239 (add_library):
  No SOURCES given to target: tbb_static

Shouldn't the latest versions of CMake 'compile' assembly out of the box?

[peterjc123]

Well, I see. It is a mis-typing in the CMake scripts. Since we use Ninja in CI, it seems to be ignored. Would you please change the code in pytorch/aten/src/ATen/cpu/tbb/CMakeLists.txt:L144-145 and see whether it works.

      list(APPEND tbb_src "${TBB_ROOT_DIR}/src/tbb/ia32-masm/atomic_support.asm"
        "${TBB_ROOT_DIR}/src/tbb/ia32-masm/itsx.asm src/tbb/ia32-masm/lock_byte.asm")
# -> to the following
      list(APPEND tbb_src "${TBB_ROOT_DIR}/src/tbb/ia32-masm/atomic_support.asm"
        "${TBB_ROOT_DIR}/src/tbb/ia32-masm/itsx.asm"
        "${TBB_ROOT_DIR}/src/tbb/ia32-masm/lock_byte.asm")

[peterjc123]

@franchenstein But it should not happen. I have tested the MSVC build using CI only a few time ago. What version of CI are you using? And can you located the assembler found using CMAKE?

 -- Using python found in C:\Jenkins\Miniconda3\python.exe
 -- The ASM_MASM compiler identification is MSVC
 -- Found assembler: C:/Program Files (x86)/Microsoft Visual Studio/2017/Community/VC/Tools/MSVC/14.11.25503/bin/Hostx86/x64/ml64.exe   <- this one
 -- Performing Test SUPPORTS_VOLATILE_FLAG

[franchenstein]

I made the proposed change and the previous error did not occur anymore. On the other hand, after more than 40 minutes, the compilation failed with the following message:

c:\users\dfranch\downloads\pytorch-scripts\pytorch\aten\src\aten\cpu\vec256\vec256_int.h(129): error C3861: '_mm256_ext
ract_epi64': identifier not found [C:\Users\dfranch\Downloads\pytorch-scripts\pytorch\aten\build\src\ATen\ATen.vcxproj]
c:\users\dfranch\downloads\pytorch-scripts\pytorch\aten\src\aten\cpu\vec256\vec256_int.h(130): error C3861: '_mm256_ext
ract_epi64': identifier not found [C:\Users\dfranch\Downloads\pytorch-scripts\pytorch\aten\build\src\ATen\ATen.vcxproj]
c:\users\dfranch\downloads\pytorch-scripts\pytorch\aten\src\aten\cpu\vec256\vec256_int.h(131): error C3861: '_mm256_ext
ract_epi64': identifier not found [C:\Users\dfranch\Downloads\pytorch-scripts\pytorch\aten\build\src\ATen\ATen.vcxproj]
c:\users\dfranch\downloads\pytorch-scripts\pytorch\aten\src\aten\cpu\vec256\vec256_int.h(132): error C3861: '_mm256_ext
ract_epi64': identifier not found [C:\Users\dfranch\Downloads\pytorch-scripts\pytorch\aten\build\src\ATen\ATen.vcxproj]
c:\users\dfranch\downloads\pytorch-scripts\pytorch\aten\src\aten\cpu\vec256\vec256_int.h(134): error C3861: '_mm256_ext
ract_epi64': identifier not found [C:\Users\dfranch\Downloads\pytorch-scripts\pytorch\aten\build\src\ATen\ATen.vcxproj]
c:\users\dfranch\downloads\pytorch-scripts\pytorch\aten\src\aten\cpu\vec256\vec256_int.h(135): error C3861: '_mm256_ext
ract_epi64': identifier not found [C:\Users\dfranch\Downloads\pytorch-scripts\pytorch\aten\build\src\ATen\ATen.vcxproj]
c:\users\dfranch\downloads\pytorch-scripts\pytorch\aten\src\aten\cpu\vec256\vec256_int.h(136): error C3861: '_mm256_ext
ract_epi64': identifier not found [C:\Users\dfranch\Downloads\pytorch-scripts\pytorch\aten\build\src\ATen\ATen.vcxproj]
c:\users\dfranch\downloads\pytorch-scripts\pytorch\aten\src\aten\cpu\vec256\vec256_int.h(137): error C3861: '_mm256_ext
ract_epi64': identifier not found [C:\Users\dfranch\Downloads\pytorch-scripts\pytorch\aten\build\src\ATen\ATen.vcxproj]
  ReduceOpsKernel.cpp.AVX2.cpp
c:\users\dfranch\downloads\pytorch-scripts\pytorch\aten\src\aten\cpu\vec256\vec256_int.h(129): error C3861: '_mm256_ext
ract_epi64': identifier not found [C:\Users\dfranch\Downloads\pytorch-scripts\pytorch\aten\build\src\ATen\ATen.vcxproj]
c:\users\dfranch\downloads\pytorch-scripts\pytorch\aten\src\aten\cpu\vec256\vec256_int.h(130): error C3861: '_mm256_ext
ract_epi64': identifier not found [C:\Users\dfranch\Downloads\pytorch-scripts\pytorch\aten\build\src\ATen\ATen.vcxproj]
c:\users\dfranch\downloads\pytorch-scripts\pytorch\aten\src\aten\cpu\vec256\vec256_int.h(131): error C3861: '_mm256_ext
ract_epi64': identifier not found [C:\Users\dfranch\Downloads\pytorch-scripts\pytorch\aten\build\src\ATen\ATen.vcxproj]
c:\users\dfranch\downloads\pytorch-scripts\pytorch\aten\src\aten\cpu\vec256\vec256_int.h(132): error C3861: '_mm256_ext
ract_epi64': identifier not found [C:\Users\dfranch\Downloads\pytorch-scripts\pytorch\aten\build\src\ATen\ATen.vcxproj]
c:\users\dfranch\downloads\pytorch-scripts\pytorch\aten\src\aten\cpu\vec256\vec256_int.h(134): error C3861: '_mm256_ext
ract_epi64': identifier not found [C:\Users\dfranch\Downloads\pytorch-scripts\pytorch\aten\build\src\ATen\ATen.vcxproj]
c:\users\dfranch\downloads\pytorch-scripts\pytorch\aten\src\aten\cpu\vec256\vec256_int.h(135): error C3861: '_mm256_ext
ract_epi64': identifier not found [C:\Users\dfranch\Downloads\pytorch-scripts\pytorch\aten\build\src\ATen\ATen.vcxproj]
c:\users\dfranch\downloads\pytorch-scripts\pytorch\aten\src\aten\cpu\vec256\vec256_int.h(136): error C3861: '_mm256_ext
ract_epi64': identifier not found [C:\Users\dfranch\Downloads\pytorch-scripts\pytorch\aten\build\src\ATen\ATen.vcxproj]
c:\users\dfranch\downloads\pytorch-scripts\pytorch\aten\src\aten\cpu\vec256\vec256_int.h(137): error C3861: '_mm256_ext
ract_epi64': identifier not found [C:\Users\dfranch\Downloads\pytorch-scripts\pytorch\aten\build\src\ATen\ATen.vcxproj]
  Generating Code...
Done Building Project "C:\Users\dfranch\Downloads\pytorch-scripts\pytorch\aten\build\src\ATen\ATen.vcxproj" (default ta
rgets) -- FAILED.

Done Building Project "C:\Users\dfranch\Downloads\pytorch-scripts\pytorch\aten\build\ALL_BUILD.vcxproj" (default target
s) -- FAILED.

Done Building Project "C:\Users\dfranch\Downloads\pytorch-scripts\pytorch\aten\build\INSTALL.vcxproj" (default targets)
 -- FAILED.

I used Cl version 19.11.25548.2 for x64. I couldn't find the assembler being used. I had the following message:

-- Using python found in C:\Users\dfranch\AppData\Local\Continuum\anaconda3\python.exe
-- Configuring done
-- Generating done
-- Build files have been written to: C:/Users/dfranch/Downloads/pytorch-scripts/pytorch/aten/build

[peterjc123]

How could _mm256_extract_epi64 be not found in MSVC 2017? It should the issue only for 2015. Can you give me your full build log?

[franchenstein]

build_log.txt

The log is quite long because of lots repeated messages. I am attaching it to this message.

[peterjc123]

Sorry, a flag is missing there. I just solved it. Could you please try again?

[franchenstein]

I added the flag but I still get the same 16 errors.

[peterjc123]

Would you please clean the build first? And make sure you call it in a simple CMD window, not a dev shell.

[franchenstein]

DId that and now everything works great! Thank you very much for the support!

[biztechprogramming]

Which flag is missing? I get the same error compiling v0.4.1 from source.

[biztechprogramming]

I found a commit on April 23, 2018 which had this added:
set CMAKE_GENERATOR=Visual Studio 15 2017 Win64

I ran that command and the errors have gone away. I don't know which file in pytorch should be updated. I had to compile the pytorch code from source, because my gpu is a GTX 960. I'm not sure if cuda 8 from these scripts is too advanced for my gpu.