build fail with cuda: sparse_xent_op.h

[WeiTang114]

OS: Debian Stretch 64
Cuda: 7.5
Cudnn: 6.5
Compiler: gcc-4.9 (default gcc-5)
Bazel: 0.1.4

Since I had a problem using gcc-5 building (see at the bottom), I tried forcing it to use gcc-4.9 by editing third_party/gpus/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc.

Now the error is about sparse_xent_op.h.

Please help, thank you.

 $ bazel build -s  -j 1  -c opt  --config=cuda //tensorflow/cc:tutorials_example_trainer                                                                                                                                            [0:28:58]
WARNING: Output base '/auto/master04/weitang114/.cache/bazel/_bazel_weitang114/043c55a899c217c377c622f5ff5a22ca' is on NFS. This may lead to surprising failures and undetermined behavior.
WARNING: Sandboxed execution is not supported on your system and thus hermeticity of actions cannot be guaranteed. See http://bazel.io/docs/bazel-user-manual.html#sandboxing for more information. You can turn off this warning via --ignore
_unsupported_sandboxing.
INFO: Found 1 target...
>>>>> # //tensorflow/core:gpu_kernels [action 'Compiling tensorflow/core/kernels/sparse_xent_op_gpu.cu.cc']
(cd /auto/master04/weitang114/.cache/bazel/_bazel_weitang114/043c55a899c217c377c622f5ff5a22ca/tensorflow && \
  exec env - \
    PATH=/home/master/04/weitang114//bin/:/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/local/cuda-7.5/bin:/home/master/04/weitang114//bin \
  third_party/gpus/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -fPIE -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 -DNDEBUG
-ffunction-sections -fdata-sections '-std=c++11' -iquote . -iquote bazel-out/local_linux-opt/genfiles -iquote external/bazel_tools -iquote bazel-out/local_linux-opt/genfiles/external/bazel_tools -iquote external/jpeg_archive -iquote bazel
-out/local_linux-opt/genfiles/external/jpeg_archive -iquote external/png_archive -iquote bazel-out/local_linux-opt/genfiles/external/png_archive -iquote external/re2 -iquote bazel-out/local_linux-opt/genfiles/external/re2 -iquote external
/eigen_archive -iquote bazel-out/local_linux-opt/genfiles/external/eigen_archive -isystem third_party/gpus/cuda/include -isystem bazel-out/local_linux-opt/genfiles/third_party/gpus/cuda/include -isystem external/bazel_tools/tools/cpp/gcc3
 -isystem google/protobuf/src -isystem bazel-out/local_linux-opt/genfiles/google/protobuf/src -isystem external/jpeg_archive/jpeg-9a -isystem bazel-out/local_linux-opt/genfiles/external/jpeg_archive/jpeg-9a -isystem external/png_archive/l
ibpng-1.2.53 -isystem bazel-out/local_linux-opt/genfiles/external/png_archive/libpng-1.2.53 -isystem external/re2 -isystem bazel-out/local_linux-opt/genfiles/external/re2 -isystem third_party/eigen3 -isystem bazel-out/local_linux-opt/genf
iles/third_party/eigen3 -isystem external/eigen_archive/eigen-eigen-b45554449873 -isystem bazel-out/local_linux-opt/genfiles/external/eigen_archive/eigen-eigen-b45554449873 -isystem third_party/gpus/cuda -isystem bazel-out/local_linux-opt
/genfiles/third_party/gpus/cuda -x cuda '-DGOOGLE_CUDA=1' '-nvcc_options=relaxed-constexpr' -no-canonical-prefixes -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -fno-canonical-sy
stem-headers '-frandom-seed=bazel-out/local_linux-opt/bin/tensorflow/core/_objs/gpu_kernels/tensorflow/core/kernels/sparse_xent_op_gpu.cu.o' -MD -MF bazel-out/local_linux-opt/bin/tensorflow/core/_objs/gpu_kernels/tensorflow/core/kernels/s
parse_xent_op_gpu.cu.d -c tensorflow/core/kernels/sparse_xent_op_gpu.cu.cc -o bazel-out/local_linux-opt/bin/tensorflow/core/_objs/gpu_kernels/tensorflow/core/kernels/sparse_xent_op_gpu.cu.o)
INFO: From Compiling tensorflow/core/kernels/sparse_xent_op_gpu.cu.cc:
./tensorflow/core/kernels/sparse_xent_op.h(58): error: explicit type is missing ("int" assumed)

./tensorflow/core/kernels/sparse_xent_op.h(58): error: expected a ">"

./tensorflow/core/kernels/sparse_xent_op.h(58): error: expected a type specifier

./tensorflow/core/kernels/sparse_xent_op.h(73): error: explicit type is missing ("int" assumed)

./tensorflow/core/kernels/sparse_xent_op.h(73): error: expected a ">"

./tensorflow/core/kernels/sparse_xent_op.h(90): error: explicit type is missing ("int" assumed)

./tensorflow/core/kernels/sparse_xent_op.h(90): error: expected a ">"

./tensorflow/core/kernels/sparse_xent_op.h(90): error: expected a type specifier

./tensorflow/core/kernels/sparse_xent_op.h(104): error: explicit type is missing ("int" assumed)

./tensorflow/core/kernels/sparse_xent_op.h(104): error: expected a ">"

./tensorflow/core/kernels/sparse_xent_op.h(122): error: identifier "int64" is undefined

./tensorflow/core/kernels/sparse_xent_op.h(133): error: identifier "int64" is undefined

./tensorflow/core/kernels/sparse_xent_op.h(185): error: no instance of constructor "tensorflow::generator::SparseXentLossGenerator<T>::SparseXentLossGenerator [with T=float]" matches the argument list
            argument types are: (Eigen::TensorMap<Eigen::Tensor<const float, 2, 1, int>, 16>, Eigen::TensorMap<Eigen::Tensor<const float, 1, 1, int>, 16>, <error-type>)
          detected during:
            instantiation of "void tensorflow::functor::SparseXentEigenImpl<Device, T>::Compute(const Device &, tensorflow::TTypes<T, 1, Eigen::DenseIndex>::ConstMatrix, tensorflow::TTypes<<error-type>, 1, Eigen::DenseIndex>::ConstVec, te
nsorflow::TTypes<T, 1, Eigen::DenseIndex>::Vec, tensorflow::TTypes<T, 1, Eigen::DenseIndex>::Vec, tensorflow::TTypes<T, 1, Eigen::DenseIndex>::Matrix) [with Device=tensorflow::GPUDevice, T=float]"
tensorflow/core/kernels/sparse_xent_op_gpu.cu.cc(39): here
            instantiation of "void tensorflow::functor::SparseXentFunctor<tensorflow::GPUDevice, T>::operator()(const tensorflow::GPUDevice &, tensorflow::TTypes<T, 1, Eigen::DenseIndex>::ConstMatrix, tensorflow::TTypes<tensorflow::int64,
 1, Eigen::DenseIndex>::ConstVec, tensorflow::TTypes<T, 1, Eigen::DenseIndex>::Vec, tensorflow::TTypes<T, 1, Eigen::DenseIndex>::Vec, tensorflow::TTypes<T, 1, Eigen::DenseIndex>::Matrix) [with T=float]"
tensorflow/core/kernels/sparse_xent_op_gpu.cu.cc(46): here

./tensorflow/core/kernels/sparse_xent_op.h(193): error: no instance of constructor "tensorflow::generator::SparseXentGradGenerator<T>::SparseXentGradGenerator [with T=float]" matches the argument list
            argument types are: (Eigen::TensorMap<Eigen::Tensor<const float, 2, 1, int>, 16>, Eigen::TensorMap<Eigen::Tensor<const float, 1, 1, int>, 16>, <error-type>)
          detected during:
            instantiation of "void tensorflow::functor::SparseXentEigenImpl<Device, T>::Compute(const Device &, tensorflow::TTypes<T, 1, Eigen::DenseIndex>::ConstMatrix, tensorflow::TTypes<<error-type>, 1, Eigen::DenseIndex>::ConstVec, te
nsorflow::TTypes<T, 1, Eigen::DenseIndex>::Vec, tensorflow::TTypes<T, 1, Eigen::DenseIndex>::Vec, tensorflow::TTypes<T, 1, Eigen::DenseIndex>::Matrix) [with Device=tensorflow::GPUDevice, T=float]"
tensorflow/core/kernels/sparse_xent_op_gpu.cu.cc(39): here
            instantiation of "void tensorflow::functor::SparseXentFunctor<tensorflow::GPUDevice, T>::operator()(const tensorflow::GPUDevice &, tensorflow::TTypes<T, 1, Eigen::DenseIndex>::ConstMatrix, tensorflow::TTypes<tensorflow::int64,
 1, Eigen::DenseIndex>::ConstVec, tensorflow::TTypes<T, 1, Eigen::DenseIndex>::Vec, tensorflow::TTypes<T, 1, Eigen::DenseIndex>::Vec, tensorflow::TTypes<T, 1, Eigen::DenseIndex>::Matrix) [with T=float]"
tensorflow/core/kernels/sparse_xent_op_gpu.cu.cc(46): here

tensorflow/core/kernels/sparse_xent_op_gpu.cu.cc(39): error: no suitable user-defined conversion from "Eigen::TensorMap<Eigen::Tensor<const tensorflow::int64, 1, 1, Eigen::DenseIndex>, 16>" to "Eigen::TensorMap<Eigen::Tensor<const <error-
type>, 1, 1, Eigen::DenseIndex>, 16>" exists
          detected during instantiation of "void tensorflow::functor::SparseXentFunctor<tensorflow::GPUDevice, T>::operator()(const tensorflow::GPUDevice &, tensorflow::TTypes<T, 1, Eigen::DenseIndex>::ConstMatrix, tensorflow::TTypes<tens
orflow::int64, 1, Eigen::DenseIndex>::ConstVec, tensorflow::TTypes<T, 1, Eigen::DenseIndex>::Vec, tensorflow::TTypes<T, 1, Eigen::DenseIndex>::Vec, tensorflow::TTypes<T, 1, Eigen::DenseIndex>::Matrix) [with T=float]"
(46): here

./tensorflow/core/kernels/sparse_xent_op.h(91): error: identifier "labels" is undefined

16 errors detected in the compilation of "/tmp/tmpxft_00007ac3_00000000-10_sparse_xent_op_gpu.cu.compute_52.cpp1.ii".
ERROR: /auto/master04/weitang114/tensorflow/tensorflow/core/BUILD:321:1: output 'tensorflow/core/_objs/gpu_kernels/tensorflow/core/kernels/sparse_xent_op_gpu.cu.o' was not created.
ERROR: /auto/master04/weitang114/tensorflow/tensorflow/core/BUILD:321:1: not all outputs were created.
Target //tensorflow/cc:tutorials_example_trainer failed to build

##

(For the details how I changed to gcc-4.9, in case I did something wrong)
At first I used gcc-5(by default), nvcc compiling core/kernels/bias_op_gpu.cu.cc:

/usr/lib/gcc/x86_64-linux-gnu/5/include/mwaitxintrin.h(36): error: identifier "__builtin_ia32_monitorx" is undefined.
/usr/lib/gcc/x86_64-linux-gnu/5/include/mwaitxintrin.h(42): error: identifier "__builtin_ia32_mwaitx" is undefined.

Then I followed #8 to modify crosstool_wrapper_driver_is_not_gcc into

 46 CURRENT_DIR = os.path.dirname(sys.argv[0])
 47 CPU_COMPILER = ('/usr/bin/gcc-4.9')
 48 NVCC_PATH = CURRENT_DIR + '/../../../cuda/bin/nvcc'
 49 GCC_HOST_COMPILER_PATH = ('/usr/bin/gcc-4.9')
 50 LLVM_HOST_COMPILER_PATH = ('/usr/bin/gcc-4.9')
 51 PREFIX_DIR = os.path.dirname(GCC_HOST_COMPILER_PATH)

[vrv]

Our GPU build is broken -- probably sync back to Thursday and you should be alright until we fix it.

[ebrevdo]

Fix is incoming.

[vrv]

Just pushed the fix

[WeiTang114]

Thank you very much!

[nlgranger]

The bug still seems to remain as of ed6f783:

INFO: From Compiling tensorflow/core/kernels/adjust_contrast_op_gpu.cu.cc:

# Omitting warnings

/usr/lib/gcc/x86_64-unknown-linux-gnu/5.3.0/include/mwaitxintrin.h(36): error: identifier "__builtin_ia32_monitorx" is undefined

/usr/lib/gcc/x86_64-unknown-linux-gnu/5.3.0/include/mwaitxintrin.h(42): error: identifier "__builtin_ia32_mwaitx" is undefined

2 errors detected in the compilation of "/tmp/tmpxft_00002a59_00000000-7_adjust_contrast_op_gpu.cu.cpp1.ii".

Build process is as follows:

  TF_UNOFFICIAL_SETTING=1 ./configure <<EOF
/usr/bin/python
y
7.5
/opt/cuda
4.0.7
/opt/cuda
5.2
EOF

    bazel build --jobs 4 --config=cuda -c opt --verbose_failures \
        --define=D_MWAITXINTRIN_H_INCLUDED=1 \
        --define=__STRICT_ANSI__=1 \
        //tensorflow/tools/pip_package:build_pip_package

Environnement is:

• bazel 0.1.5-1
• cuda 7.5
• cudnn 4
• gcc 5.3.0
• boost 1.60 (is it even relevant?)
• python3 (/usr/bin/python is python3)

I also tried without the extra flags.

[vrv]

Most likely the flags for you are passing for STRICT_ANSI etc aren't making it to nvcc. You might need to edit the crosstool file in third_party/gpus/ to pass the flags there.

[nlgranger]

Indeed...
Thank you very much for this indication.
The following patch seems to fix the issue, althought I am honestly not quite sure about what these flags exactly do:

diff --git a/third_party/gpus/crosstool/CROSSTOOL b/third_party/gpus/crosstool/CROSSTOOL
index dfde7cd..547441f 100644
--- a/third_party/gpus/crosstool/CROSSTOOL
+++ b/third_party/gpus/crosstool/CROSSTOOL
@@ -46,6 +46,8 @@ toolchain {
   # Use "-std=c++11" for nvcc. For consistency, force both the host compiler
   # and the device compiler to use "-std=c++11".
   cxx_flag: "-std=c++11"
+  cxx_flag: "-D_MWAITXINTRIN_H_INCLUDED"
+  cxx_flag: "-D__STRICT_ANSI__"
   linker_flag: "-lstdc++"
   linker_flag: "-B/usr/bin/"

Could anyone kindly reopen this bug until this patch gets reviewed, and integrated (or not) please?

[vrv]

I would argue this belongs in a new issue, the existing bug was for a separate problem.