crosstool_wrapper_driver_is_not_gcc failed: error executing command

[drscotthawley]

Please go to Stack Overflow for help and support:

http://stackoverflow.com/questions/tagged/tensorflow

If you open a GitHub issue, here is our policy:

1. It must be a bug or a feature request. This is a bug
2. The form below must be filled out.

Here's why we have that policy: TensorFlow developers respond to issues. We want to focus on work that benefits the whole community, e.g., fixing bugs and adding features. Support only helps individuals. GitHub also notifies thousands of people when issues are filed. We want them to see you communicating an interesting problem, rather than being redirected to Stack Overflow.

---

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 16.04
• TensorFlow installed from (source or binary): source
• TensorFlow version (use command below): n/a, GitHub clone, 11:12pm CST May 3 2017.
• Bazel version (if compiling from source): 0.4.5
• CUDA/cuDNN version: 8.0.61
• GPU model and memory: NVIDIA Titan X Pascal (12GB)
• Exact command to reproduce: bazel build --config=opt --config=cuda //tensorflow/tools/pip_package:build_pip_package --verbose_failures

You can collect some of this information using our environment capture script:

https://github.com/tensorflow/tensorflow/tree/master/tools/tf_env_collect.sh

You can obtain the TensorFlow version with

python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"

Describe the problem

Describe the problem clearly here. Be sure to convey here why it's a bug in TensorFlow or a feature request.

This is a bug relating to installation. I'm getting a similar error to #336 (tensorflow/serving#336) except without any reference to nccl, similar to #8709 except I'm on Ubuntu 16.04 instead of 14, similar to #8790 (#8790) except that one seems to have been closed prematurely.

Source code / logs

Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem.

$ bazel build --config=opt --config=cuda //tensorflow/tools/pip_package:build_pip_package --verbose_failures
.......
WARNING: /home/anaconda/tensorflow/tensorflow/contrib/learn/BUILD:15:1: in py_library rule //tensorflow/contrib/learn:learn: target '//tensorflow/contrib/learn:learn' depends on deprecated target '//tensorflow/contrib/session_bundle:exporter': Use SavedModel Builder instead.
WARNING: /home/anaconda/tensorflow/tensorflow/contrib/learn/BUILD:15:1: in py_library rule //tensorflow/contrib/learn:learn: target '//tensorflow/contrib/learn:learn' depends on deprecated target '//tensorflow/contrib/session_bundle:gc': Use SavedModel instead.
INFO: Found 1 target...
ERROR: /home/anaconda/tensorflow/tensorflow/core/BUILD:1299:1: C++ compilation of rule '//tensorflow/core:lib_hash_crc32c_accelerate_internal' failed: crosstool_wrapper_driver_is_not_gcc failed: error executing command 
  (cd /home/anaconda/.cache/bazel/_bazel_anaconda/8174be50bc0029f23c89ddd1973ff713/execroot/tensorflow && \
  exec env - \
    CUDA_TOOLKIT_PATH=/usr/local/cuda \
    CUDNN_INSTALL_PATH=/usr/local/cuda-8.0 \
    GCC_HOST_COMPILER_PATH=/usr/bin/gcc \
    PYTHON_BIN_PATH=/opt/anaconda/envs/py27/bin/python \
    PYTHON_LIB_PATH=/opt/anaconda/envs/py27/lib/python2.7/site-packages \
    TF_CUDA_CLANG=0 \
    TF_CUDA_COMPUTE_CAPABILITIES=6.1 \
    TF_CUDA_VERSION='' \
    TF_CUDNN_VERSION='' \
    TF_NEED_CUDA=1 \
    TF_NEED_OPENCL=0 \
  external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -fPIE -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 -DNDEBUG -ffunction-sections -fdata-sections '$opt' '-std=c++11' '$opt' -MD -MF bazel-out/local_linux-opt/bin/tensorflow/core/_objs/lib_hash_crc32c_accelerate_internal/tensorflow/core/lib/hash/crc32c_accelerate.pic.d '-frandom-seed=bazel-out/local_linux-opt/bin/tensorflow/core/_objs/lib_hash_crc32c_accelerate_internal/tensorflow/core/lib/hash/crc32c_accelerate.pic.o' -fPIC -iquote . -iquote bazel-out/local_linux-opt/genfiles -iquote external/bazel_tools -iquote bazel-out/local_linux-opt/genfiles/external/bazel_tools -isystem external/bazel_tools/tools/cpp/gcc3 -DEIGEN_AVOID_STL_ARRAY -Iexternal/gemmlowp -Wno-sign-compare -fno-exceptions '-DGOOGLE_CUDA=1' -msse3 -pthread -msse4.2 -no-canonical-prefixes -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -fno-canonical-system-headers -c tensorflow/core/lib/hash/crc32c_accelerate.cc -o bazel-out/local_linux-opt/bin/tensorflow/core/_objs/lib_hash_crc32c_accelerate_internal/tensorflow/core/lib/hash/crc32c_accelerate.pic.o): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1.
gcc: error: $opt: No such file or directory
gcc: error: $opt: No such file or directory
Target //tensorflow/tools/pip_package:build_pip_package failed to build
INFO: Elapsed time: 15.639s, Critical Path: 6.51s
$


$ bazel version
Build label: 0.4.5
Build target: bazel-out/local-fastbuild/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Thu Mar 16 12:19:38 2017 (1489666778)
Build timestamp: 1489666778
Build timestamp as int: 1489666778

$ cat /usr/local/cuda/version.txt 
CUDA Version 8.0.61

$ gcc --version
gcc (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

On Ubuntu 16.04, building for Python 2.7 (from Anaconda). Running from bash.
I grabbed tensorflow from GitHub tonight, accepted all the the defaults when running configure, said yes CUDA support (more defaults) and used compute capability 6.1.
Tried bazel clean, re-configure, etc...same result.

I've built this successfully for my Python 3.5 environment on this computer a couple months ago, just trying to add a Python 2.7 version and now it's failing.

[StatML]

I meet the same problem. Hope it could be solved ASAP.

[fvisin]

Same here.

[skye]

@gunan can you advise? From the error output, it looks like there's an environment variable (?) $opt that's not defined.

[shamoya]

Same here.
This what git blame says:
$ git show 22586bd configure
.
.

+echo >> tools/bazel.rc                                   
 for opt in $CC_OPT_FLAGS; do                             
-  write_to_bazelrc 'build:opt --cxxopt=$opt --copt=$opt' 
+  echo "build:opt --cxxopt=$opt --copt=$opt" >> tools/bazel.rc
 done 

If I revert this code the compilation succeeds.

[drpngx]

@nlopezgi could you comment?

[drpngx]

Actually, it looks like we are using single quotes which doesn't interpolate, rather than double quotes previously.

Fixing internally. Should be in the next push.

[drscotthawley]

Confirmed. Changing from single quotes to double quotes on line 357 of configure fixed it for me.

For others: make that change in configure (or wait for the push by @drpngx ), re-run configure, then do bazel build...everything works from there. :-)

[nyoshimura]

Hi, I faced same problem when I tried to run the command on Jetson TX2 by following this instruction.

But when I changed the parameter of command from 3072 to 1800 by following another instruction, it worked well :)

The root cause may be ran out of memory.