Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[build fail] python: can't open '....': [Errno2] No such file or directory #3788

Open
sp4ke opened this issue Sep 22, 2017 · 5 comments
Open

[build fail] python: can't open '....': [Errno2] No such file or directory #3788

sp4ke opened this issue Sep 22, 2017 · 5 comments

Comments

@sp4ke
Copy link

@sp4ke sp4ke commented Sep 22, 2017

Please provide the following information. The more we know about your system and use case, the more easily and likely we can help.

Description of the problem / feature request / question:

Trying to build tensorflow with opencl support using this repo.
The bazel build is giving me the following error
python: can't open file 'external/local_config_sycl/crosstool/computecpp': [Errno 2] No such file or directory

Here's a verbose output:

ERROR: /home/spike/.cache/bazel/_bazel_spike/ed14cddfddd6d6c9f1aad38aca9f1862/external/farmhash_archive/BUILD.bazel:19:1: C++ compilation of rule '@farmhash_archive//:farmhash' failed (Exit 2): computecpp failed: error executing command 
  (cd /home/spike/.cache/bazel/_bazel_spike/ed14cddfddd6d6c9f1aad38aca9f1862/execroot/org_tensorflow && \
  exec env - \
    COMPUTECPP_TOOLKIT_PATH=/opt/ComputeCpp-CE-0.3.1-Linux \
    HOST_CXX_COMPILER=/usr/bin/g++ \
    HOST_C_COMPILER=/usr/bin/gcc \
    PATH=/home/spike/.nvm/versions/node/v6.10.1/bin:/home/spike/.local/bin:/home/spike/bin:/home/spike/.pyenv/plugins/pyenv-virtualenv/shims:/home/spike/.pyenv/shims:/home/spike/scripts:/home/spike/projects/go:/home/spike/projects/go/bin:/home/spike/.local/bin:/home/spike/bin:/home/spike/.pyenv/plugins/pyenv-virtualenv/shims:/home/spike/.pyenv/shims:/home/spike/scripts:/home/spike/projects/go:/home/spike/projects/go/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl \
    PWD=/proc/self/cwd \
    PYTHON_BIN_PATH=/home/spike/.pyenv/versions/ml/bin/python \
    PYTHON_LIB_PATH=/home/spike/.pyenv/versions/ml/lib/python3.5/site-packages \
    TF_NEED_CUDA=0 \
    TF_NEED_OPENCL=1 \
    TF_VECTORIZE_SYCL=1 \
  external/local_config_sycl/crosstool/computecpp -fPIE -fno-omit-frame-pointer -Wall -msse3 -g0 -O2 -DNDEBUG -ffunction-sections -fdata-sections '-march=native' '-std=c++11' '-march=native' -MD -MF bazel-out/local_linux-py3-opt/bin/external/farmhash_archive/_objs/farmhash/external/farmhash_archive/src/farmhash.pic.d '-frandom-seed=bazel-out/local_linux-py3-opt/bin/external/farmhash_archive/_objs/farmhash/external/farmhash_archive/src/farmhash.pic.o' -fPIC -iquote external/farmhash_archive -iquote bazel-out/local_linux-py3-opt/genfiles/external/farmhash_archive -iquote external/bazel_tools -iquote bazel-out/local_linux-py3-opt/genfiles/external/bazel_tools -isystem external/farmhash_archive/src -isystem bazel-out/local_linux-py3-opt/genfiles/external/farmhash_archive/src -isystem external/bazel_tools/tools/cpp/gcc3 -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -fno-canonical-system-headers -c external/farmhash_archive/src/farmhash.cc -o bazel-out/local_linux-py3-opt/bin/external/farmhash_archive/_objs/farmhash/external/farmhash_archive/src/farmhash.pic.o).
python: can't open file 'external/local_config_sycl/crosstool/computecpp': [Errno 2] No such file or directory
Target //tensorflow/tools/pip_package:build_pip_package failed to build
INFO: Elapsed time: 0.313s, Critical Path: 0.08s

When I check inside /home/spike/.cache/bazel/_bazel_spike/ed14cddfddd6d6c9f1aad38aca9f1862/execroot/org_tensorflow the path external/local_config_sycl/crosstool/computecpp exists and I can run it from the command line and I can even compile the file manually using the command that is run by bazel from the same working directory.

Inside the external path I changed all symlinks to real files but still the same error.

How can I debug the python command that is run by bazel in this case ?

If possible, provide a minimal example to reproduce the problem:

Any bazel build --config=sycl command

Environment info

  • Operating System: Arch Linux 64

  • Bazel version (output of bazel info release):
    both release 0.5.4- (@non-git) and development version (HEAD)

  • If bazel info release returns "development version" or "(@non-git)", please tell us what source tree you compiled Bazel from; git commit hash is appreciated (git rev-parse HEAD):

Have you found anything relevant by searching the web? No

(e.g. StackOverflow answers,
GitHub issues,
email threads on the bazel-discuss Google group)

Anything else, information or logs or outputs that would be helpful?

(If they are large, please upload as attachment or provide link).

@damienmg

This comment has been minimized.

Copy link
Contributor

@damienmg damienmg commented Sep 22, 2017

Sounds like sandboxing so @philwo to take a stab at it.

@Taltzipi

This comment has been minimized.

Copy link

@Taltzipi Taltzipi commented Oct 19, 2017

I Ran into a similar issue compiling tensorflow using a pyenv python version, where I received

python: can't open file 'external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc': [Errno 2] No such file or directory

It seems there's a clash in the use of pyenv and bazel.

What worked for me was:

  1. temporarily disable pyenv (commenting out its initialization in .bashrc and opening a new bash) so that 'which python' points to the system's /usr/bin/python,
  2. run tensorflow's configure.py. When asked for the python version, point it to a specific pyenv python version you've installed, eg ~/.pyenv/versions/3.6.1/bin/python
  3. continue with the bazel build - it should now run successfully.
  4. reenable pyenv in bashrc

There's probably a more elegant way, but worth giving this a try.

@philwo

This comment has been minimized.

Copy link
Member

@philwo philwo commented Oct 19, 2017

Thanks for your suggestion @Taltzipi! I can imagine that this helps.

I don't think this is a sandboxing bug. Usually these kind of issues result from the environment being somehow "special" (e.g. Python is installed via pyenv, from source, relies on LD_LIBRARY_PATH or PATH being set to special values...).

Not saying that this shouldn't work with Bazel, but it means that I also don't have any more clue what the reason is than anyone else on the team ;)

It might help to consider that the error "[Errno 2] No such file or directory" can also (a bit confusingly) mean that the binary relies on shared libraries that were not found, which could be, because Bazel strips certain environment variables when running actions?

@vishnubob

This comment has been minimized.

Copy link

@vishnubob vishnubob commented Jan 9, 2018

I am building tensorflow from scratch for the first time, and I immediately hit this issue. I did not want to restart the whole process over again without my carefully constructed pyenv environment, so I looked for a solution. Since the file external/local_config_sycl/crosstool/computecpp is located within our build cache, my (gross) fix was to add a new line to the top of this python script that points directly to the pyenv python executable I wish to run.

(echo "#\!$(pyenv which python)" && cat /PATH/TO/crosstool_wrapper_driver_is_not_gcc) > /tmp/cross
mv /tmp/cross /PATH/TO/crosstool_wrapper_driver_is_not_gcc

I believe the "problem" is the builders' usage of env - which dumps the environment before it hands the subshell over to external/local_config_sycl/crosstool/computecpp via exec. Since we are in a naked environment, I would assume the pyenv shims aren't properly initialized, and the shell reports that it can't find python.

@philwo

This comment has been minimized.

Copy link
Member

@philwo philwo commented Jul 19, 2018

Unassigning - sorry, I can't be of help fixing this. It's not really a sandboxing issue (in the sense of: we could change Bazel's sandbox somehow to make this work). I think pyenv might be just fundamentally incompatible with our sandboxing - but that said, I don't know anything about pyenv. I think someone with more experience in Python should look into this.

@philwo philwo removed their assignment Jul 19, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.