Error with building clang unknown argument

[eugeny-stoyka]

I configured project for building and execute next command in project directory:

bazel build -c opt --config=cuda //tensorflow/cc:tutorials_example_trainer

After that i get this error:

ERROR: /home/eugeny/Git/tensorflow/tensorflow/core/kernels/BUILD:3614:1: C++ compilation of rule '//tensorflow/core/kernels:multinomial_op_gpu' failed: clang failed: error executing command /usr/bin/clang -MD -MF bazel-out/local_linux-py3-opt/bin/tensorflow/core/kernels/_objs/multinomial_op_gpu/tensorflow/core/kernels/multinomial_op_gpu.cu.d ... (remaining 142 argument(s) skipped): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1.
clang: error: unknown argument: '-nvcc_options=relaxed-constexpr'
clang: error: unknown argument: '-nvcc_options=ftz=true'
clang: error: cannot find libdevice for sm_35. Provide path to different CUDA installation via --cuda-path, or pass -nocudalib to build without linking with libdevice.
clang: error: cannot find CUDA installation. Provide its path via --cuda-path, or pass -nocudainc to build without CUDA includes.
clang: error: cannot find libdevice for sm_52. Provide path to different CUDA installation via --cuda-path, or pass -nocudalib to build without linking with libdevice.
clang: error: cannot find CUDA installation. Provide its path via --cuda-path, or pass -nocudainc to build without CUDA includes.
clang: error: cannot find CUDA installation. Provide its path via --cuda-path, or pass -nocudainc to build without CUDA includes.
Target //tensorflow/cc:tutorials_example_trainer failed to build

I'm trying to build it with gpu support by using next libs:

Bazel: 0.5.1
gcc: 7.1.1
Cuda: 8 (version of cuda/bin/gcc is 5.4.0)
CUDNN: 6
protobuf: 3.3.1

What shall i change for normal building?

[Randl]

I've got the same error with current version of clang (clang-5.0).

[Randl]

It looks like clang doesn't support those nvcc flags, at least I haven't found information on it anywhere, including docs

[eugeny-stoyka]

Yeah, i tried it with clang3.5 and clang4, noway to build tf with them.

[Randl]

@skye Why it's closed?

[eugeny-stoyka]

I just asked for versions of building tools and no answer for it. Pretty sad you cant help me with it, because i have no clue how deal with it.

[skye]

Ah sorry, didn't realize you were still asking for help.

Can you fill out the new issue template please? In particular what OS are you using?

@gunan it looks like clang/gcc versions aren't documented, do you know if/when we expect building with clang to work?

[gunan]

Are you configuring to use when building for GPU?
You may need a bleeding edge build of clang, that has GPU support.

Also, could you fill in the issue template?
I cannot really comment much without knowing your OS version, what commit you guys synced to, etc.

[Randl]

@gunan I've used Ubuntu 16.04, clang-5.0 (i.e. development branch), and master branch of tensorflow. CUDA 8.0 and cudnn 6.

[gunan]

@Randl, which version of TensorFlow are you trying to build?
Also, when configuring, are you trying to use clang as the cuda compiler?
Or is clang simply the default compiler on your system while you use nvcc for GPU code.

[Randl]

@gunan
I'm trying to build 708cbaf
I've configured clang as CUDA compiler and I'm using gcc as system compiler and Tensorflow compiler.

[gunan]

@ilya-biryukov Could you look into this issue?
Which version of clang do we need to use to build TF with GPU support?

Maybe we should add a note in the configure script for the minimum required clang version.

[eugeny-stoyka]

Hello, thank you all for respond. I use archlinux and same build configuration as @Randl, but i can easily switch into ubuntu 16.04

[eugeny-stoyka]

@gunan is it possible to create docker container for building tensorflow somehow?

[Randl]

I'm using clang version 5.0.0-svn306258-1~exp1 and it does compile CUDA code (I've tried with a couple of toy examples).

[gunan]

@4l4l4l our default builds use gcc and nvcc to build.
There are already docker containers for tensorflow for those, you can use any "devel" docker container in our dockerhub.

clang is still work in progress, so Ill have to wait for a response from @ilya-biryukov to see what the problem you are running into is.

[ilya-biryukov]

Please use --config=cuda_clang after configuring to build using clang. (I also think that using bazel build without --config flags should work too).
We should add a check for that and provide a helpful error message.

[ilya-biryukov]

The build currently also fails because of an error in configure script. The fix should be upstream soon, the workaround before it lands is to change the following lines of .tf_configure.bazelrc from:

build --config=cuda
test --config=cuda

to:

build --config=cuda_clang
test --config=cuda_clang

This should be done after running configure.

[Randl]

@ilya-biryukov Thanks, with those two fixes I managed to build Tensorflow

[skye]

Thanks for the update @Randl! I'm gonna close this, but @4l4l4l please reopen if Ilya's advice doesn't work for you. Thanks.

[eugeny-stoyka]

@ilya-biryukov my build in progress but i get next error

[eugeny@localhost tensorflow]$ bazel build -c opt //tensorflow/cc:tutorials_example_trainer
INFO: Found 1 target...
ERROR: /home/eugeny/Git/tensorflow/tensorflow/core/kernels/BUILD:3036:1: C++ compilation of rule '//tensorflow/core/kernels:depth_space_ops_gpu' failed: clang failed: error executing command /usr/bin/clang -MD -MF bazel-out/local_linux-py3-opt/bin/tensorflow/core/kernels/_objs/depth_space_ops_gpu/tensorflow/core/kernels/spacetodepth_op_gpu.cu.d ... (remaining 141 argument(s) skipped): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1.
clang: error: cannot find libdevice for sm_50. Provide path to different CUDA installation via --cuda-path, or pass -nocudalib to build without linking with libdevice.
clang: error: cannot find CUDA installation.  Provide its path via --cuda-path, or pass -nocudainc to build without CUDA includes.
clang: error: cannot find CUDA installation.  Provide its path via --cuda-path, or pass -nocudainc to build without CUDA includes.
Target //tensorflow/cc:tutorials_example_trainer failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 1.632s, Critical Path: 0.23s

Where should i insert --cuda-path flag?

I found solution. Guys, you should work better with configuration files. I directly install my cuda sources in /opt/cuda and set it in configuration but your code still seeks it in /usr/local/cuda path somewhere

thank you @ilya-biryukov for help and other guys too, all works well

[skye]

Thank you for sharing your solution @4l4l4l!

[ilya-biryukov]

@4l4l4l, the configure script is not very good at figuring out cuda install paths other than /usr/local/cuda. And it should also probably do some sanity-checking (i.e. at least check that the cuda folder exists).

Anyway, glad you figured it out.