ERROR: no such package '@local_config_cuda//crosstool': BUILD file not found on package path. #4105
Comments
trevorwelch
changed the title
|
I got this recently too, I was somehow successful by just re-running ./configure and then immediately running bazel build, but I'm not sure what's going on. |
|
Following that @vrv I just tried de-installing TF completely and then re-running with the same ./configure settings: Then a simplified build command: After a very verbose and lengthy compile attempt, I received this error message (it caused OS X Terminal app to hang permanently as well, so I couldn't copy-paste, had to take a screenshot): https://www.dropbox.com/s/riu5f4n5aj1opmk/Screenshot%202016-08-30%2016.33.48.png?dl=0 Yet again, some sort of a |
|
Yes if you are developing TF it happens quite often. I think it might be something related with the caching system of bazel. After few hours it resets and I have to re run the configure again. |
|
+cc @damienmg This seems to be a bug in Bazel. (Edit: to clarify: I meant the occasional Next time this happens, can you take a look at directory |
|
@davidzchen thanks for your reply. This error or some version of it is consistent: |
|
I just saw your screenshot, and that appears to be a different problem than the Do you mean that the |
|
I meant that my TF builds seem to fail related to crosstool consistently, probably my naivete on the specifics for me to think that |
|
No problem. The naming could be a bit confusing. The Were you able to reproduce the |
|
I am getting the same missing crosstool on Linux. The strange thing is that there isn't even a bazel-tensorflow directory: bazel is 0.3.1, and I have ran ./configure four times now. |
|
I do see that Update: Same error message related to The complete log was too big for pastebin, here it is on Dropbox: https://www.dropbox.com/home/Documents%20Dropbox?preview=TW-TF-error-log-083116.txt Then, I run the same above commands, but with --verbose_failures (hard to imagine it being more verbose that the previous log, which was almost 15,000 lines!), final error message was: Here's the complete log: https://www.dropbox.com/s/nozqcscnc9ho5uz/TW-TF-error-log-083116--verbose_failures.txt?dl=0 |
|
@Dapid That is interesting. How did you run the @damienmg Is there currently a way to inspect the contents of @trevorwelch FWIW, most of the noise in the output are compiler warnings. The |
On branch r0.10. If I leave the default GCC it is created, but the build fails because it is incompatible with CUDA. |
|
@davidzchen |
|
Whoa, I'm running into this, too, but on master with OS X 10.11.6. ./configure: Here's my file listings. All of the symlinks work and the files are all Mach-O so no weird accidental ELF or something: https://gist.github.com/jmhodges/a5de9cc5760333f5b57040d1947ec190 This was after going to sleep and coming back to this just now. Last night, I was debugging a different error condition and just came back to find my builds no longer working. I thought it was me hand-hacking in extra linkopts ( |
|
I can confirm this also building on a Debian (sid, uptodate of today) system. |
|
I've found I can induce this by Ctrl-C'ing in the middle of a fresh |
|
Ubuntu-16.04, CUDA 8, java 1.8.0_101, bazel 0.3.1 Building from master today Started ./configure in a virtual instance in VirtualBox, did a CTRL-C because it was taking too long. Went home, fired up the instance again, deleted tensorflow repo, cloned it again. Did ./configure again with same options as before, it worked well except one warning at the beginning: Ignored it, and did the command to build for GPU: And it failed immediately: EDIT: Tried |
|
There are two issues being discussed in this thread. @trevorwelch, let's move the For those experiencing the
In the meantime, I am still trying to reproduce this. |
|
i experience the same issue. tf 0.10. mac os el capitain. bazel 0.3.0 |
|
@asimonov - Can you print the contents of the |
contents of bazel cache after BUILD file not found Here's how to reproduce it: Clone this repo: https://github.com/FlorinAndrei/ml-setup Checkout the ubuntu1604 branch, then launch the virtual machine and run the ansible installer, then compile TF by hand: However, if you delete the tensorflow repo, re-clone and try again, it starts compiling: contents of bazel cache after ./configure And now it starts compiling. EDIT: Even on second try, it still fails to compile all the way to the end, but that seems like a different issue, which I've opened here: |
|
David, I cannot find local_config_cuda/crosstool directory anywhere in tensorflow directory. Kind Regards,
|
|
@davidzchen At Maybe worth noting that there does exist a And so the symlink to it at |
|
Got this problem ERROR: no such package '@local_config_cuda//crosstool': again today.
|
|
meet the same issue. |
* Run bazel clean and bazel fetch in the configure script even when building without GPU support to force clean+fetch if the user re-runs ./configure with a different setting. * Print a more actionable error messsage if the user attempts to build with --config=cuda but did not configure TensorFlow to build with GPU support. * Update the BUILD file in @local_config_cuda to use repository-local labels. Fixes tensorflow#4105
|
Hi. I'm getting "ERROR: no such package '@local_config_cuda//crosstool': BUILD file not found on package path." as well with:
The issue for me happens deterministically, if I run tensorflow ./configure while trying to avoid interactive questions: set vars to avoid interactiveexport PYTHON_BIN_PATH=/usr/bin/python No way to confirm the following default value to ./util/python/python_config.sh without actually hitting the Return key: :-((/usr/lib/python3/dist-packagesexport TF_NEED_GCP=n If I run ./configure without setting the above variables, ie.: ubuntu@aws17:~/tensorflow$ ./configure /usr/local/lib/python3.5/dist-packages then the compilation works. Note that there is probably no way to pass blank values (indicating "use the default" as opposed to undefined values indicating "I have not answered yet") for several of the variables. So ./configure in the interactive mode is getting various things blank while the less interactive ./configure has these values filled. |
* Run bazel clean and bazel fetch in the configure script even when building without GPU support to force clean+fetch if the user re-runs ./configure with a different setting. * Print a more actionable error messsage if the user attempts to build with --config=cuda but did not configure TensorFlow to build with GPU support. * Update the BUILD file in @local_config_cuda to use repository-local labels. Fixes tensorflow#4105
* Run bazel clean and bazel fetch in the configure script even when building without GPU support to force clean+fetch if the user re-runs ./configure with a different setting. * Print a more actionable error messsage if the user attempts to build with --config=cuda but did not configure TensorFlow to build with GPU support. * Update the BUILD file in @local_config_cuda to use repository-local labels. Fixes tensorflow#4105
…4285) * Run bazel clean and bazel fetch in the configure script even when building without GPU support to force clean+fetch if the user re-runs ./configure with a different setting. * Print a more actionable error messsage if the user attempts to build with --config=cuda but did not configure TensorFlow to build with GPU support. * Update the BUILD file in @local_config_cuda to use repository-local labels. Fixes #4105
|
@martinwicke A similar error now appears during the building process: I think this might be related to 4316aeb |
|
Hi, Facing similar issues for tensorflow build. b) ERROR: I/O error while writing action log: No space left on device. Enviornment: Cuda 8.0 Tried ./configure and build several times. Configure is successful but build fails. Also tried, bazel clean, bazel clean --explunge and ran the build with reduced number of jobs but the error continues.. Looked at this thread and also Here is the full error message: 52adb8ea4f53b1b72067611e8a7eb020/external/grpc/BUILD:69:1: C++ compilation of rule '@grpc//:gpr' failed: crosstool_wrapper_driver_is_not_gcc failed: error executing command external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -fPIE -Wall -Wunused-but-set-parameter ... (remaining 38 argument(s) skipped): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1. |
|
@sskgit - This looks like b) no space left on device is the reason for a) compilation of a file fails. You'll notice the |
|
@darrengarvey Thanks for your response. I tried sudo df and here is the output df It shows mounted on / has almost no space, and I think $VM/Username (i.e. $HOME) is on the same mount /, as there is no $HOME mounted on, in the output. Is this the right command? At / (used space) du -sch At $HOME (used space) du -sch So, I am not sure what is occupying the remaining space? Total storage on this VM is 380 GB. Is there any way to get rid of bazel logs? Is it causing the space issue? If so, where? Thanks |
|
@sskgit Running @martinwicke Sorry for the late reply. I have been on call for a good part of the past week. That looks like a linker error in protobuf and does not seem related to this particular change. |
|
@davidzchen Thanks for your response. used bazel clean --expunge (this cleaned 500MB space) and re-configured tensorflow using ./configure Run the bazel build again.. bazel build -c opt --config=cuda //tensorflow/tools/pip_package:build_pip_package ERROR: /$HOME/Downloads/tensorflow/tensorflow/core/kernels/BUILD:1710:1: error while parsing .d file: /$HOME/.cache/bazel/_bazel_gpuadmin/52adb8ea4f53b1b72067611e8a7eb020/execroot/tensorflow/bazel-out/local_linux-opt/bin/tensorflow/core/kernels/_objs/depth_space_ops_gpu/tensorflow/core/kernels/depthtospace_op_gpu.cu.pic.d (No such file or directory). At $HOME: At /: df -h This is a fairly new machine with few installer software (<200MB) |
|
@sskgit Interesting. Can you open a bug at https://github.com/bazelbuild/bazel for this issue? Thanks. |
|
@davidzchen Opened an issue with bazel as well. I have tried the build almost 10 times in last couple of days. Still trying to figure why the build fails and how do I complete the build successfully. |
…ensorflow#4285) * Run bazel clean and bazel fetch in the configure script even when building without GPU support to force clean+fetch if the user re-runs ./configure with a different setting. * Print a more actionable error messsage if the user attempts to build with --config=cuda but did not configure TensorFlow to build with GPU support. * Update the BUILD file in @local_config_cuda to use repository-local labels. Fixes tensorflow#4105
|
Space issue caused my Tensorflow build to fail. Clearing some space on the / mount made the build successful and Tensorflow now works as expected. Thanks everyone for your help! |
|
This is happening for me too. I already have TF installed and working for GPU via the runfile but I wanted to compile it for optimizations. I get: Steps to reproduce: If you notice that I've done something wrong, please let me know. I saw someone before mention something about |
|
I’m also facing the same challenge - unable to build tensor flow on a gpu server. Below given are the details. OS is Ubuntu 16.04LTS |
|
Also -when checked the cache doesn’t contain crosstool - Am I missing something here? |
|
@itssujeeth #11949 fixes the issue when building tensorflow with gpu support using bazel 0.5.3. |
and others
Environment info
Operating System:
OS 10.10.5
Installed version of CUDA and cuDNN:
python -c "import tensorflow; print(tensorflow.__version__)".(can't get that far, but i'm using 0.10)
If installed from source, provide
git rev-parse HEAD)bazel versionIf possible, provide a minimal reproducible example (We usually don't have time to read hundreds of lines of your code)
What other attempted solutions have you tried?
./configureto not use symlinked librariesThe text was updated successfully, but these errors were encountered: