enable C++ extension on Windows #1345

carolineechen · 2021-03-03T18:32:06Z

after making sox and kaldi selective (#1338, #1342), we can enable C++ extension to be built on Windows without requiring bindings to either of them

torchaudio/backend/sox_io_backend.py

torchaudio/extension/extension.py

build_tools/setup_helpers/extension.py

carolineechen · 2021-03-03T23:25:02Z

@mthrok to get the cmake compiler to work, I need to set an environment variable to "the full path to the compiler, or to the compiler name if it is in the PATH". Do you know how to figure out this path or name on Windows, or if it's something I'd need to additionally set up as part of the build?

mthrok · 2021-03-04T03:31:51Z

@mthrok to get the cmake compiler to work, I need to set an environment variable to "the full path to the compiler, or to the compiler name if it is in the PATH". Do you know how to figure out this path or name on Windows, or if it's something I'd need to additionally set up as part of the build?

Can you try this and copy the helper script from here?

mthrok · 2021-03-04T03:32:42Z

Also we might need to do git submodule update --init --recursive in windows installation script.

build_tools/setup_helpers/extension.py

mthrok · 2021-03-04T22:53:08Z

For the sake of debugging, can you add '-DCMAKE_EXPORT_COMPILE_COMMANDS=ON',, and then print the content of 'compile_commands.json', so that we can compare the command between Python and see what's different?

See
https://github.com/pytorch/text/blob/master/build_tools/setup_helpers/extension.py#L122-L124

mthrok · 2021-03-05T03:08:10Z

Comparing the link command log, looks like "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python37_64\libs\python37.lib" is being placed where the correct value is supposed to be ..\..\..\env\libs\python36.lib.

I think this explains why we get LINK : fatal error LNK1104: cannot open file 'python36.lib' error. It's not provided to the linker while torch_python.lib is linked to python36.lib.

I guess that this means that find_package(Torch REQUIRED) fills the wrong value of ${TORCH_LIBRARIES}.

@carolineechen Can you print out the values for these and check the hypothesis?
Also can you print the content of env/libs directory before starting the build? add ls env/libs in .circleci/unittest/windows/scripts/install.sh before running python setup.py install.

Successful link from unittest_windows_cpu_py3.7

[5/6] cmd.exe /C "cd . &&
C:\Users\circleci\project\env\Lib\site-packages\cmake\data\bin\cmake.exe
 -E vs_link_dll
 --intdir=torchaudio\csrc\CMakeFiles\_torchaudio.dir
 --rc=C:\PROGRA~2\WI3CF2~1\10\bin\100190~1.0\x64\rc.exe
 --mt=C:\PROGRA~2\WI3CF2~1\10\bin\100190~1.0\x64\mt.exe
 --manifests
 --
 C:\PROGRA~2\MICROS~2\2019\COMMUN~1\VC\Tools\MSVC\1428~1.293\bin\Hostx64\x64\link.exe
 torchaudio\csrc\CMakeFiles\_torchaudio.dir\pybind.cpp.obj
 torchaudio\csrc\CMakeFiles\_torchaudio.dir\lfilter.cpp.obj
 torchaudio\csrc\CMakeFiles\_torchaudio.dir\overdrive.cpp.obj
 torchaudio\csrc\CMakeFiles\_torchaudio.dir\utils.cpp.obj
 /out:torchaudio\csrc\_torchaudio.dll
 /implib:torchaudio\csrc\_torchaudio.lib
 /pdb:torchaudio\csrc\_torchaudio.pdb
 /dll /version:0.0
 /machine:x64 /INCREMENTAL:NO
  ..\..\..\env\Lib\site-packages\torch\lib\c10.lib
  ..\..\..\env\Lib\site-packages\torch\lib\torch_python.lib
  ..\..\..\env\libs\python37.lib <------------------------------------------- correct python library
  ..\..\..\env\Lib\site-packages\torch\lib\torch.lib
  ..\..\..\env\Lib\site-packages\torch\lib\torch_cpu.lib
  ..\..\..\env\Lib\site-packages\torch\lib\c10.lib
  kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib&& cd ."

Unsuccessful link from unittest_windows_cpu_py3.6

[5/6] cmd.exe /C "cd . &&
C:\Users\circleci\project\env\Lib\site-packages\cmake\data\bin\cmake.exe
 -E vs_link_dll
 --intdir=torchaudio\csrc\CMakeFiles\_torchaudio.dir
 --rc=C:\PROGRA~2\WI3CF2~1\10\bin\100190~1.0\x64\rc.exe
 --mt=C:\PROGRA~2\WI3CF2~1\10\bin\100190~1.0\x64\mt.exe
 --manifests
  --
 C:\PROGRA~2\MICROS~2\2019\COMMUN~1\VC\Tools\MSVC\1428~1.293\bin\Hostx64\x64\link.exe
  torchaudio\csrc\CMakeFiles\_torchaudio.dir\pybind.cpp.obj
 torchaudio\csrc\CMakeFiles\_torchaudio.dir\lfilter.cpp.obj
 torchaudio\csrc\CMakeFiles\_torchaudio.dir\overdrive.cpp.obj
 torchaudio\csrc\CMakeFiles\_torchaudio.dir\utils.cpp.obj
  /out:torchaudio\csrc\_torchaudio.dll
 /implib:torchaudio\csrc\_torchaudio.lib
 /pdb:torchaudio\csrc\_torchaudio.pdb
 /dll /version:0.0 
/machine:x64 /INCREMENTAL:NO
  ..\..\..\env\Lib\site-packages\torch\lib\c10.lib
  ..\..\..\env\Lib\site-packages\torch\lib\torch_python.lib
  "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python37_64\libs\python37.lib" <------ wrong python library
  ..\..\..\env\Lib\site-packages\torch\lib\torch.lib
  ..\..\..\env\Lib\site-packages\torch\lib\torch_cpu.lib
  ..\..\..\env\Lib\site-packages\torch\lib\c10.lib
  kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib  && cd ."

mthrok · 2021-03-05T17:07:15Z

With commit 0b76f5f (workflow)

binary_windows_wheel_py3.6
-- Found Python3: C:/Program Files (x86)/Microsoft Visual Studio/Shared/Python37_64/libs/python37.lib (found version "3.7.8") found components: Development Development.Module Development.Embed
binary_windows_wheel_py3.7
-- Found Python3: C:/tools/miniconda3/envs/env3.7/libs/python37.lib (found version "3.7.10") found components: Development Development.Module Development.Embed
binary_windows_wheel_py3.8
-- Found Python3: C:/tools/miniconda3/envs/env3.8/libs/python38.lib (found version "3.8.8") found components: Development Development.Module Development.Embed
binary_windows_wheel_py3.9
-- Found Python3: C:/tools/miniconda3/envs/env3.9/libs/python39.lib (found version "3.9.1") found components: Development Development.Module Development.Embed

mthrok · 2021-03-05T17:13:17Z

@carolineechen

Can you try find_package(Python3 COMPONENTS Interpreter Development)?

According to https://gitlab.kitware.com/cmake/cmake/-/issues/19024#note_534082, this will force the find_package to find the library that is compatible with Python interpreter. If the correct interpreter is found, then there is a good chance that the matching library is found.

mthrok

Looks good. Thanks!

* Rename torchtext_translation_tutorial.py to torchtext_translation.py Temporarily disable execution. * Update index.rst Temp link update for torchtext_translation tutorial.

enable windows extension

a805f3b

facebook-github-bot added the CLA Signed label Mar 3, 2021

install cmake ninja for windows env

6332cb6

mthrok reviewed Mar 3, 2021

View reviewed changes

torchaudio/backend/sox_io_backend.py Show resolved Hide resolved

mthrok reviewed Mar 3, 2021

View reviewed changes

torchaudio/extension/extension.py Outdated Show resolved Hide resolved

mthrok reviewed Mar 3, 2021

View reviewed changes

build_tools/setup_helpers/extension.py Show resolved Hide resolved

carolineechen changed the title ~~enable C++ extension on Windows~~ [WIP] enable C++ extension on Windows Mar 3, 2021

revert minimal changes, add windows extension settings

e803c9c

carolineechen force-pushed the windows_extension branch from 44df6f0 to e803c9c Compare March 3, 2021 20:59

mthrok mentioned this pull request Mar 4, 2021

Add backprop support to lfilter #1310

Merged

mthrok reviewed Mar 4, 2021

View reviewed changes

build_tools/setup_helpers/extension.py Outdated Show resolved Hide resolved

carolineechen force-pushed the windows_extension branch 7 times, most recently from ed79c28 to 5b62fc6 Compare March 4, 2021 18:50

windows env

05a2241

carolineechen force-pushed the windows_extension branch from 5b62fc6 to 05a2241 Compare March 4, 2021 19:09

remove extra changes

3bd4512

mthrok added the module: windows label Mar 4, 2021

use cmake args

e0c0ab9

add debug output

07ca402

find PYTHON_VERSION first

0b76f5f

add Interpreter

8dbc94d

carolineechen force-pushed the windows_extension branch 4 times, most recently from be915fd to 2c38237 Compare March 5, 2021 19:32

remove interpreter, add vs2019, debug

75d388c

carolineechen force-pushed the windows_extension branch from 2c38237 to 75d388c Compare March 5, 2021 19:39

remove debug

8c57d00

mthrok approved these changes Mar 5, 2021

View reviewed changes

carolineechen changed the title ~~[WIP] enable C++ extension on Windows~~ enable C++ extension on Windows Mar 5, 2021

carolineechen merged commit 9395ad6 into pytorch:master Mar 5, 2021

carolineechen deleted the windows_extension branch March 5, 2021 21:02

This was referenced Mar 6, 2021

Experimental windows builds #219

Closed

Fix C++ extension initialization for Windows #1371

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

enable C++ extension on Windows #1345

enable C++ extension on Windows #1345

carolineechen commented Mar 3, 2021

carolineechen commented Mar 3, 2021

mthrok commented Mar 4, 2021

mthrok commented Mar 4, 2021

mthrok commented Mar 4, 2021

mthrok commented Mar 5, 2021 •

edited

mthrok commented Mar 5, 2021

mthrok commented Mar 5, 2021

mthrok left a comment

enable C++ extension on Windows #1345

enable C++ extension on Windows #1345

Conversation

carolineechen commented Mar 3, 2021

carolineechen commented Mar 3, 2021

mthrok commented Mar 4, 2021

mthrok commented Mar 4, 2021

mthrok commented Mar 4, 2021

mthrok commented Mar 5, 2021 • edited

mthrok commented Mar 5, 2021

mthrok commented Mar 5, 2021

mthrok left a comment

Choose a reason for hiding this comment

mthrok commented Mar 5, 2021 •

edited