Tensorflow GPU installation fails due to Bazel build

[elexira]

I've been trying to install tensorflow with GPU support using these steps:
http://www.nvidia.com/object/gpu-accelerated-applications-tensorflow-installation.html

This is the error message that I'm getting when I try to run the bazel build command for building the tensorflow pip package (with the --config-cuda flag set):

bazel build -c opt --config=cuda //tensorflow/tools/pip_package:build_pip_package

i get :
The specified --crosstool_top '//third_party/gpus/crosstool:crosstool' is not a valid cc_toolchain_suite rule

What's strange is that if i remove the --config=cuda flag, I don't get the error message while building and I'm able to install tensorflow successfully - but without GPU support.

[wagonhelm]

The posting guidelines ask for information that could be very helpful such as:

Similar Posts
Operating System
Cuda, CudNN & Bazel Version

The type of graphic card as well wouldn't hurt. I may suggest a "$ bazel clean" and re-run the "./configure" as well.

[jart]

It seems this issue is a duplicate of http://stackoverflow.com/questions/40010981/tensorflow-with-gpu-support-installation-error-the-specified-crosstool-top-i. We recommend Stack Overflow for support questions. We try to keep this issue tracker focused on bugs and feature requests. Also thanks for helping out with support @wagonhelm.

[DanlanChen]

not solved yet

[jart]

Since we're getting multiple reports, I'm going to reopen this issue so it can be triaged again.

[wagonhelm]

I normally use Ubuntu 14.04 but I had no problems installing from sources using:

Ubuntu 16.04
Cuda 8
Cudnn 5.1.5
tensorflow0.11rc1
Bazel 0.3.2

These are the steps I took, and I try to keep them up to date with each new version. I did notice a couple expected problems when trying to build and install using anaconda 3 if you just do conda install pip and use the pip packages from tf's website it works fine. I also notice a couple things that might cause issues with those Nvidia instructions.

[firolino]

The pointed out nvidia documentation (http://www.nvidia.com/object/gpu-accelerated-applications-tensorflow-installation.html) says
git reset --hard 70de76e

This commit is 4 months old! I assume this is why people need to downgrade their bazel version as well to get it work. Moreover, the documentation is using CUDA 8.0 with that commit. But 8.0 is supported as of r0.11-rc1. Guess, the nvidia documentation should be updated.

[elexira]

this issue was giving me a lot of headache. After talking to a few experienced computer vision engineers, i was told that the 'docker' installation of tensor-flow is the easiest way to get things going
https://www.tensorflow.org/versions/r0.11/get_started/os_setup.html#docker-installation

i tried docker and it saved so much of my time. I run everything on that and do not have to worry about the compatibility issues.

[asimshankar]

Glad the docker instructions worked for you @Sadrpour
As @firolino pointed out, the documentation at the NVIDIA website is asking for a very old version of the TensorFlow source code.

Updating the Nvidia website is not something the TensorFlow team can do :).
Please point this out to the Nvidia folks, and we will try to do the same. Thanks!

[vasantivmahajan]

I am facing the same issue. I updated the CROSSTOOL file and added cxx_builtin_include_directory: "/usr/local/cuda/include" But I am still facing the same issue.
I am using the following:
Ubuntu 14.04
Cuda 7.5
Cudnn 5.0
Bazel 0.4.0

Can someone guide me as to how to build GPU enabled TensorFlow?

[lan2720]

I met the similar problem. Any solution or advice? Thanks.

[lan2720]

And anyone knows how to uninstall bazel completely? I suppose it's the bazel problem and want to reinstall it.

[uher]

I think this issue is not related with bazel version.
I'm facing this issue on bazel 0.4.1 and bazel 0.3.2 also.
Anyone know how to fixed this issue?

Update. should be used bazel 0.3.2!!!

@ian2720 You can remove bazel as the following. (refer link: bazelbuild/bazel#838)
rm -fr ~/.bazel ~/.bazelrc
rm -fr ~/.cache/bazel
And you can install manually old version bazel (refer link: https://bazel.build/versions/master/docs/install.html#compiling-from-source)

[uher]

I've complete to build tensorflow.
I'm using the latest source version(Not using git reset --hard 70de76e ) after trying to build then completed.

So, try the the following command
$git reset --hard
$git pull
$git reset --hard 287db3a

The commit info is as the following:
commit 287db3a
Merge: 50460e9 52c46a7
Author: Andrew Harp andrewharp@users.noreply.github.com
Date: Tue Dec 6 23:10:35 2016 -0500
Merge pull request #6137 from andrewharp/branch_141241224
Branch 141241224

[uher]

Folks.
I've written the installation script of the GPU-enabled version of Tensorflow.
Please see the below link. :)
https://github.com/uher/InstallGpuEnableTensorflow

[dr2mer]

In my case, --config=cuda7.5 is working

[StrangeTcy]

@dr2mer : tried it your way with both --config=cuda7.5 and --config=cuda8.0. Both eventually resulted in

ERROR: /home/myname/tensorflow/tensorflow/python/BUILD:1081:1: in cc_library rule //tensorflow/python:tf_session_helper: non-test target '//tensorflow/python:tf_session_helper' depends on testonly target '//tensorflow/python:construction_fails_op' and doesn't have testonly attribute set.
ERROR: Analysis of target '//tensorflow/tools/pip_package:build_pip_package' failed; build aborted.
INFO: Elapsed time: 1.129s

The specs are Ubuntu 17.04 x64, bazel 0.4.2 CUDA 8.0 and cudnn 5.1.

[phg1024]

@StrangeTcy Commenting out testonly=1 in the target construction_fails_op resolved the issue for me.

[ShangxuanWu]

@dr2mer worked for cuda8.0. THX!

[Ettrai]

@dr2mer I followed your suggestion and the build managed to finish using --config=cuda7.5 .
The problem is that I tried to run this example but it always runs on the cpu and I have no idea why.

[Ettrai]

I am not sure that setting the flag to something other than --config=cuda gets the job actually done.
Could you double check that you're actually running your code on the gpu?

I ended up solving my problem like this : #7118

[azagovora]

I agree with @Ettrai. In my case --config=cuda8.0 or --config=cuda8 is working, but the code runs on the CPU only.

[kaltu]

I tried
bazel build --config=opt --config=cuda8.0 //tensorflow/tools/pip_package:build_pip_package
and it says

WARNING: Config values are not defined in any .rc file: cuda8.0

So apparently it just ignore the cuda8.0 option as if it was not there and therefore is equivalent to
bazel build --config=opt //tensorflow/tools/pip_package:build_pip_package
the command to build CPU only version.

[rudrivyas]

i had install tensorflow,but it show error in importing tensorflow due to bazel...! any help that i can get?

[ywu36]

In my case '--config=cuda' not working but '--config=cuda8.0' works. I am using a previous version (v0.9) to run some previous codes while v1.2 is out.

[LiberiFatali]

In my case, I use older bazel version and fix this issue