Cmake build with GPU fails on Windows 10 with a MSB6006 error

[Thimira]

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 10
• TensorFlow installed from (source or binary): Source, from Git tag v1.8.0
• TensorFlow version (use command below): 1.8 (Git tag v1.8.0)
• Python version: 3.5.5 (Anaconda Python)
• Bazel version (if compiling from source): Cmake 3.11.3
• GCC/Compiler version (if compiling from source): Visual Studio 2015 Update 3 (14.0.25431.01)
• CUDA/cuDNN version: CUDA 9.2, cuDNN 7.1
• GPU model and memory: Nvidia GeForce 940M
• Exact command to reproduce:

git clone https://github.com/tensorflow/tensorflow
....
git checkout v1.8.0
....
C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\bin\amd64\vcvarsall.bat amd64

cmake .. -A x64 -DCMAKE_BUILD_TYPE=Release ^
 -DSWIG_EXECUTABLE=C:/Dev_Tools/TFBuild/tools/swigwin-3.0.12/swig.exe ^
 -DPYTHON_EXECUTABLE=C:/Users/UAMARTH/AppData/Local/Continuum/anaconda3/envs/tfbuild/python.exe ^
 -DPYTHON_LIBRARIES=C:/Users/UAMARTH/AppData/Local/Continuum/anaconda3/envs/tfbuild/libs/python35.lib ^
 -Dtensorflow_ENABLE_GPU=ON ^
 -Dtensorflow_CUDA_VERSION=9.2 ^
 -Dtensorflow_CUDNN_VERSION=7 ^
 -DCUDNN_HOME="C:\Dev_Tools\TFBuild\tools\cudnn-9.2-windows10-x64-v7.1" ^
 -Dtensorflow_WIN_CPU_SIMD_OPTIONS=/arch:AVX


MSBuild /p:Configuration=Release tf_python_build_pip_package.vcxproj

Describe the problem

The build fails with a "error MSB6006: "cmd.exe" exited with code 1". Full error message below.
The line 259 of referred to in the error message points to a "tensorflow\tensorflow\contrib\cmake\build\tf_python\tensorflow\tools\api\generator\api\keras\preprocessing\text_init_.py" file, which does not exist. The .......\preprocessing\text\ directory is empty.

Source code / logs

"C:\Dev_Tools\TFBuild\tensorflow\tensorflow\contrib\cmake\build\tf_python_build_pip_package.vcxproj" (default target) (
1) ->
"C:\Dev_Tools\TFBuild\tensorflow\tensorflow\contrib\cmake\build\tf_python_api.vcxproj" (default target) (259) ->
(CustomBuild target) ->
  C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.CppCommon.targets(171,5): error MSB6006: "cmd.exe" e
xited with code 1. [C:\Dev_Tools\TFBuild\tensorflow\tensorflow\contrib\cmake\build\tf_python_api.vcxproj]

    19591 Warning(s)
    1 Error(s)

Time Elapsed 04:30:49.38

[angerson]

@gunan Can you take a look at this?

[gunan]

@annarev or @case540 may be able to help more on this failure. Looks like api generation failed?

[annarev]

I will look more into it but for now a random guess:
Do you have tensorflow installed with anaconda on the same machine? We saw some issue before where building TensorFlow with cmake might fail if there is existing tensorflow installation. (Basically one step in api generation would "import tensorflow" which won't be the tensorflow we just built but instead the one installed with anaconda)

[Thimira]

@annarev I have several anaconda environments setup on the machine I attempted to build TensorFlow, and one of the environments does have TensorFlow installed from pip. The anaconda environment which I attempted the build however does not have TensorFlow installed. My environment details are below,

conda info --env
# conda environments:
#
base                  *  C:\Users\UAMARTH\AppData\Local\Continuum\anaconda3
aws-tools                C:\Users\UAMARTH\AppData\Local\Continuum\anaconda3\envs\aws-tools
deep-learning            C:\Users\UAMARTH\AppData\Local\Continuum\anaconda3\envs\deep-learning
tfbuild                  C:\Users\UAMARTH\AppData\Local\Continuum\anaconda3\envs\tfbuild

From the above, the 'deep-learning' environment has TensorFlow installed from pip (tensorflow-gpu 1.5.0), while the other environments does not have it. I attempted the cmake build on the 'tfbuild' environment.

But, let me remove the environment with TensorFlow installed and re-run the build to see whether it has an effect.
Should I attempt the build with the current head commit from master branch, or should I go with a stable branch such as r1.9 or r1.8?

[Thimira]

I attempted the following:
I removed all the anaconda environments which had TensorFlow installed from the machine. I cloned a fresh copy of TensorFLow, which had the following head commit at the time of cloning,

git log
commit 359f53686c87ee76e80353c32a3d22cfb1cf0989 (HEAD -> master, origin/master, origin/HEAD)
Author: Yong Tang <yong.tang.github@outlook.com>
Date:   Thu Jun 21 22:09:56 2018 -0700

I re-ran the build using the following command (this time using the 'base' anaconda environment, which had Python 3.6),

cmake ..  -G "Visual Studio 14 2015 Win64" -T host=x64 ^
 -DCMAKE_BUILD_TYPE=Release ^
 -DSWIG_EXECUTABLE=C:/Dev_Tools/TFBuild/tools/swigwin-3.0.12/swig.exe ^
 -DPYTHON_EXECUTABLE=C:/Users/UAMARTH/AppData/Local/Continuum/anaconda3/python.exe ^
 -DPYTHON_LIBRARIES=C:/Users/UAMARTH/AppData/Local/Continuum/anaconda3/libs/python36.lib ^
 -Dtensorflow_ENABLE_GPU=ON ^
 -Dtensorflow_CUDA_VERSION=9.2 ^
 -Dtensorflow_CUDNN_VERSION=7 ^
 -DCUDNN_HOME="C:\Dev_Tools\TFBuild\tools\cudnn-9.2-windows10-x64-v7.1" ^
 -Dtensorflow_WIN_CPU_SIMD_OPTIONS=/arch:AVX

And the build failed (after 4 hours 40 minutes) with the following error,

"C:\Dev_Tools\TFBuild\tensorflow\tensorflow\contrib\cmake\build\tf_python_build_pip_package.vcxproj" (default target) (
1) ->
"C:\Dev_Tools\TFBuild\tensorflow\tensorflow\contrib\cmake\build\estimator_python_api.vcxproj" (default target) (2) ->
"C:\Dev_Tools\TFBuild\tensorflow\tensorflow\contrib\cmake\build\pywrap_tensorflow_internal.vcxproj" (default target) (3
) ->
"C:\Dev_Tools\TFBuild\tensorflow\tensorflow\contrib\cmake\build\pywrap_tensorflow_internal_static.vcxproj" (default tar
get) (4) ->
"C:\Dev_Tools\TFBuild\tensorflow\tensorflow\contrib\cmake\build\tf_c_python_api.vcxproj" (default target) (136) ->
(ClCompile target) ->
  C:\Dev_Tools\TFBuild\tensorflow\tensorflow\c\python_api.cc(19): fatal error C1083: Cannot open include file: 'tensorf
low/python/framework/cpp_shape_inference.pb.h': No such file or directory [C:\Dev_Tools\TFBuild\tensorflow\tensorflow\c
ontrib\cmake\build\tf_c_python_api.vcxproj]

    25406 Warning(s)
    1 Error(s)

Time Elapsed 04:40:36.11

I found the following issue: #18931 which points to the same error with "cpp_shape_inference.pb.h". I will attempt the fix mentioned in that issue next.

[Thimira]

The fix suggested in #18931 did not solve the problem (although, the build did run few minutes longer). The build now fails with the following error.

"C:\Dev_Tools\TFBuild\tensorflow\tensorflow\contrib\cmake\build\tf_python_build_pip_package.vcxproj" (default target) (
1) ->
"C:\Dev_Tools\TFBuild\tensorflow\tensorflow\contrib\cmake\build\estimator_python_api.vcxproj" (default target) (2) ->
(CustomBuild target) ->
  C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.CppCommon.targets(171,5): error MSB6006: "cmd.exe" e
xited with code 1. [C:\Dev_Tools\TFBuild\tensorflow\tensorflow\contrib\cmake\build\estimator_python_api.vcxproj]


"C:\Dev_Tools\TFBuild\tensorflow\tensorflow\contrib\cmake\build\tf_python_build_pip_package.vcxproj" (default target) (
1) ->
"C:\Dev_Tools\TFBuild\tensorflow\tensorflow\contrib\cmake\build\tf_python_api.vcxproj" (default target) (264) ->
  C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.CppCommon.targets(171,5): error MSB6006: "cmd.exe" e
xited with code 1. [C:\Dev_Tools\TFBuild\tensorflow\tensorflow\contrib\cmake\build\tf_python_api.vcxproj]

    25792 Warning(s)
    2 Error(s)

Time Elapsed 04:47:42.89

[annarev]

Sorry for not replying earlier. To answer your earlier question, any version is ok to try for building.
Do you see any other error in the log? The last one doesn't give much information. Can you try searching for "error :" or ": error" (don't remember which one is the right one)?

[Thimira]

@annarev I didn't see any other errors in the output. But let me re-run the build with 'detailed' verbosity and see if it shows any errors.

[Thimira]

@annarev I re-ran the build with detailed verbosity with the latest code from the master branch as of today (26/6/18 - 91780a3). The build still fails, but now it gives a different error.

"C:\Dev_Tools\TFBuild\tensorflow\tensorflow\contrib\cmake\build\tf_python_build_pip_package.vcxproj" (default target) (
1) ->
"C:\Dev_Tools\TFBuild\tensorflow\tensorflow\contrib\cmake\build\estimator_python_api.vcxproj" (default target) (2) ->
"C:\Dev_Tools\TFBuild\tensorflow\tensorflow\contrib\cmake\build\pywrap_tensorflow_internal.vcxproj" (default target) (3
) ->
"C:\Dev_Tools\TFBuild\tensorflow\tensorflow\contrib\cmake\build\pywrap_tensorflow_internal_static.vcxproj" (default tar
get) (4) ->
"C:\Dev_Tools\TFBuild\tensorflow\tensorflow\contrib\cmake\build\tf_stream_executor.vcxproj" (default target) (148) ->
(ClCompile target) ->
  C:\Dev_Tools\TFBuild\tensorflow\tensorflow/core/platform/test.h(34): fatal error C1083: Cannot open include file: 'gt
est/gtest.h': No such file or directory (compiling source file C:\Dev_Tools\TFBuild\tensorflow\tensorflow\stream_execut
or\lib\statusor_test.cc) [C:\Dev_Tools\TFBuild\tensorflow\tensorflow\contrib\cmake\build\tf_stream_executor.vcxproj]

    25383 Warning(s)
    1 Error(s)

Time Elapsed 04:46:45.42

I did a search for "error:" in the build log as you suggested, which returned the following lines,

Testing ReaderTest/parseWithOneError: OK
Testing ReaderTest/parseChineseWithOneError: OK
Testing ReaderTest/parseWithDetailError: OK

": error" didn't have any matches.

Please find the complete build log here: msbuild.log. Please note that the log file is about 100MB.

[gunan]

For the latest issue, I just pushed a fix to master. Could you try pulling head and retrying?

[Thimira]

@gunan I re-attempted the build, by cloning a fresh copy (with the head commit at bfcfad5). However, it still failed with the following error,

"C:\Dev_Tools\TFBuild\tensorflow\tensorflow\contrib\cmake\build\tf_python_build_pip_package.vcxproj" (default target) (
1) ->
"C:\Dev_Tools\TFBuild\tensorflow\tensorflow\contrib\cmake\build\estimator_python_api.vcxproj" (default target) (2) ->
(CustomBuild target) ->
  C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.CppCommon.targets(171,5): error MSB6006: "cmd.exe" e
xited with code 1. [C:\Dev_Tools\TFBuild\tensorflow\tensorflow\contrib\cmake\build\estimator_python_api.vcxproj]


"C:\Dev_Tools\TFBuild\tensorflow\tensorflow\contrib\cmake\build\tf_python_build_pip_package.vcxproj" (default target) (
1) ->
"C:\Dev_Tools\TFBuild\tensorflow\tensorflow\contrib\cmake\build\tf_python_api.vcxproj" (default target) (264) ->
  C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.CppCommon.targets(171,5): error MSB6006: "cmd.exe" e
xited with code 1. [C:\Dev_Tools\TFBuild\tensorflow\tensorflow\contrib\cmake\build\tf_python_api.vcxproj]

    25797 Warning(s)
    2 Error(s)

Time Elapsed 04:50:43.32

Please find the full build log here.

Question: does the build depend on any python modules already being installed? Can already installed python modules conflict with it? I have verified that tensorflow is not installed anywhere in my machine before building. But, I do have protobuf, and absl-py installed. Can it cause an issue?

[RomainDWipsea]

Hi, I have the exact same errors, here is my config if that helps :

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 10
• TensorFlow installed from (source or binary): Source
• TensorFlow version (use command below): 1.9 (commit 80fb867)
• Python version: 3.5.4rc1
• Bazel version (if compiling from source): Cmake 3.10.0rc4
• GCC/Compiler version (if compiling from source): Visual Studio 2015 Update 3 (14.0.25431.01)
• CUDA/cuDNN version: CUDA 9.0, cuDNN 7.0
• GPU model and memory: Nvidia GTX 1070
• Exact command to reproduce:
  cmake .. -G "Visual Studio 14 2015 Win64" -DCMAKE_BUILD_TYPE=Release ^
  -DSWIG_EXECUTABLE=C:\Users\dambr\wipsea\logiciels\swigwin-3.0.12\swigwin-3.0.12\swig.exe ^
  -DPYTHON_EXECUTABLE=C:\Users\dambr\AppData\Local\Programs\Python\Python35\python.exe ^
  -DPYTHON_LIBRARIES=C:\Users\dambr\AppData\Local\Programs\Python\Python35\libs\python35.lib ^
  -Dtensorflow_ENABLE_GPU=ON ^
  -DCUDNN_HOME="C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0" ^
  -DCUDA_TOOLKIT_ROOT_DIR="C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0"

MSBuild /m:4 /p:Configuration=Release /p:PreferredToolArchitecture=x64 tensorflow.sln
(run as admin)

And the resulting errors :

   "C:\Users\dambr\code\tensorflow\tensorflow\contrib\cmake\build\tensorflow.sln" (default target) (1) ->
   "C:\Users\dambr\code\tensorflow\tensorflow\contrib\cmake\build\tf_python_api.vcxproj.metaproj" (default target)
   (257) ->
   "C:\Users\dambr\code\tensorflow\tensorflow\contrib\cmake\build\tf_python_api.vcxproj" (default target) (530) ->
   (CustomBuild target) ->
     C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.CppCommon.targets(171,5): error MSB6006: "cmd
   .exe" exited with code 1. [C:\Users\dambr\code\tensorflow\tensorflow\contrib\cmake\build\tf_python_api.vcxproj]


   "C:\Users\dambr\code\tensorflow\tensorflow\contrib\cmake\build\tensorflow.sln" (default target) (1) ->
   "C:\Users\dambr\code\tensorflow\tensorflow\contrib\cmake\build\estimator_python_api.vcxproj.metaproj" (default t
   arget) (238) ->
   "C:\Users\dambr\code\tensorflow\tensorflow\contrib\cmake\build\estimator_python_api.vcxproj" (default target) (5
   28) ->
     C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\V140\Microsoft.CppCommon.targets(171,5): error MSB6006: "cmd
   .exe" exited with code 1. [C:\Users\dambr\code\tensorflow\tensorflow\contrib\cmake\build\estimator_python_api.vc
   xproj]

102 Warning(s)
2 Error(s)

[LoSealL]

For windows and cmake buiding, there is an issue when generating python api, in the scipt create_python_api.py. See my response to #20669 .

[RomainDWipsea]

Thank you @LoSealL , uninstalling tensorflow-gpu and grpcio (pip uninstall) according to your comment on #20669 solved the issue for me. I was able compile without errors and run Release/tf_tutorials_example_trainer.exe as advised for checking there.

PS : I first tried to remove just tensorflow-gpu, but it failed. It lead to a new error MSB6006 raised by the _beam_search_ops project.

[tensorflowbutler]

Nagging Assignee @gunan: It has been 59 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

[gunan]

I think this was fixed, but I forgot to update this issue.
However, now this is obsolete because the recommended way to build TF on windows is now using bazel:
https://www.tensorflow.org/install/install_sources_windows

As a bonus, building TF on windows with GPU support is 5x faster using bazel than cmake!