New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Installing Tensorflow with GPU support on os x 10.11 #2940
Comments
Hey guys I actually pass the Anyone has any idear? |
You have to run bazel-bin/tensorflow/tools/pip_package/build_pip_package.
|
Hi Martin: Thanks to reply this, really. I run 'bazel-bin/tensorflow/tools/pip_package/build_pip_package' but got a I spent a day googled it and some said I need to install the wheel and There is a closed issue: #348 Any ideas? Martin Wicke notifications@github.com于2016年6月23日周四 上午11:32写道:
|
I ran into a similar issue with Python 2.7.11 on OS X 10.11.5 + MBP. $ ./configure Setting up Cuda include To verify CUDA install: CUDA Device Query (Runtime API) version (CUDART static linking) Detected 1 CUDA Capable device(s) Device 0: "GeForce GT 650M" deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 7.5, CUDA Runtime Version = 7.5, NumDevs = 1, Device0 = GeForce GT 650M |
Hi peter: I actually pass the build process. Maybe you want set a cuda capability during the tensorflow configuration? And your error log said: And make sure your source code is updated. I got a wired error once by On Fri, Jun 24, 2016, 3:04 AM peterswang notifications@github.com wrote:
|
Thx for your suggestions, Velkan. I followed the instructions under the Build from Source section of the TF [Download and Setup] page. Compute capability was specified to be '3.0', just that the copy-paste missed that line. The Github TF page shows last successful build for Mac OS to be 14 days ago. I just tried building again with a fresh clone from the repo and it crashed at the same place. Actually, bazel apparently invoked Python, which crashed, seemingly while trying to access the CUDA library, perhaps for libcufftw.7.5.dylib? If it may be of any help to Martin, below is part of call stack from the crash report: Exception Type: EXC_BAD_ACCESS (SIGSEGV) VM Regions Near 0: Thread 0 Crashed:: Dispatch queue: com.apple.main-thread |
Just another datapoint. I'm getting the same python crash using After Python crashes for Possible useful config information:
Crash stack
|
@peterswang btw, I have seen see this
|
There's a bunch of separate issues on this thread. The half_plus_two issue should have been resolved since. @tomnielsen @Velkan as for the wheel problem -- I haven't seen that other than when there's a lingering old version of setuptools around. Can you try installing in a virtualenv? Especially on Mac that often helps. |
I also hit the same issue on Mac. I investigated a little bit. As mentioned earlier in this thread, the crash happens because tensorflow couldn't load a cuda shared library. tensorflow shouldn't crash but it did because As for why tensorflow didn't find the cuda shared library, in my case, it was because ipython from anaconda doesn't inherit the environment variable from my bash shell. I switched to use python instead of ipython, tensorflow was able to load the cuda shared library. Hopefully this finding will help others who have similar problem.
|
@Invisibility nice digging. I've seen similar looking crashes on Mac in other places with similar stack trace, now I know where to look for. BTW, you can set env var in anaconda directly using something like this
|
Thanks for the tip, @yaroslavvb. I tried, but it still crashed. It seems that for some reason
My workaround is to avoid using python.app, i.e. running the following command insteead of directly running ipython:
|
I'm also getting a segmentation fault in python.
test.py
System Info:
Stack Trace
edit: The issue occurs because the environment variable LD_LIBRARY_PATH is not set. I set it to the same as DYLD_LIBRARY_PATH which resolved the segmentation fault from a null path as discovered above in #2940 (comment) by @Invisibility . I then encountered this error:
I created a symlink to in
|
Try Python3 instead of Python2 |
I'm currently using a GTX 960 2GB at 4096x2160@60Hz. Based on #2840, I wonder if all the VRAM is used up for the display which is why I'm getting a segmentation fault? Also Chrome is using 49.18 GB / 64 GB system memory.. I have a GTX 960 4GB that I can install and test tomorrow. |
Previously I was following these instructions https://gist.github.com/Mistobaan/dd32287eeb6859c6668d and built from
I also rebuilt this using |
BTW, there's now a daily wheel with Mac GPU support:
|
There is a bug with loading libcuda.dylib - the default cuda install creates libcuda.dylib, but tensorflow tries to load libcuda.1.dylib . This fails, resorting to using the LD_LIBRARY_PATH which if NULL crashes. If you copy libcuda.dylib to libcuda.1.dylib it loads fine. |
Thanks @alexcolburn, copying libcuda.dylib to libcuda.1.dylib indeed fixed this issue on my Mac 10.10 |
I confirmed comment of @xuanchien . So a simple solution. Worked on my MBP 2014 (nvidia 750M) with Mac OS 10.12 |
Thanks. I applied the same trick and now my TF on python 2.7 on OSX 10.11.5 works. |
I'm closing this issue, I believe we have fixed the cuda version problem as well. |
The issues still persist in tensorflow 1.0.0 alpha:
alexcolburn advice to link libcuda.1.dylib to libcuda.dylib does help. |
Same problem after installing Tensorflow today. Version info:
Problem description:
After creating a symlink:
The remaining error was due to protobuf not being installed correctly. I had previously run
|
This is still a common problem on OS X 10.11.5 / 10.11.6 as we have new team members following the instructions at https://www.tensorflow.org/install/install_mac The solution is to create a link in /usr/local/cuda/lib with 'sudo ln -s /usr/local/cuda/lib/libcuda.dylib /usr/local/cuda/lib/libcuda.1.dylib' |
@victorv Unfortunately, this problem still persists. I am getting
|
Here's a successful 10.11.6 build. Mainly for my own documentation, it has been simplified and Just Works™ : Pre-requisites:OS X 10.11.6 Download and install directly from nVIDIA:
Test CUDA installCopy expected output (for my GTX 960 4GB, yours will vary depending on your GPU):
Other software installsYou might have to install I recommend
Install Tensorflow
Testtensorflow-test.py:
expected output (the final result should be the same, the only differences being the GPU used and amount of free GPU RAM):
Enjoy your tensorflow install and make something insanely great.FeedbackIf this worked or more importantly, didnt work, let me know and I'll try to keep it updated. |
@hbfs please clarify specifics of:
What do the curly brackets mean? |
Had managed to build TF 0.12 by following How to fix dyld: Library not loaded: @rpath/libcudart.7.5.dylib issue when you build tensorflow on your Mac. Just tried the building TF 1.0 from source using the same method and now the following error shows up after the patch in genrule-setup.sh: Environment: 2012 MBP with Nvidia 650M, OS X 10.12.3, Python 2.7.13. Links in /usr/local/cuda/lib, as suggested by @victorv was set before, though this doesn't seem to be the same issue. "pip install tensorflow-gpu" does work. However, the standard build apparently doesn't include AVX & SSE, which I'd like to make use of as well. |
@esd100 Do you have more detail on your segfault? |
@victorv Tell me how to get more detail and I'll happily provide it. I am using: ========================================= Files are installed in: ========================================= CUDA Device Query (Runtime API) version (CUDART static linking) Detected 1 CUDA Capable device(s) Device 0: "GeForce GT 750M" deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 8.0, CUDA Runtime Version = 8.0, NumDevs = 1, Device0 = GeForce GT 750M ========================================= $ system_profiler SPSoftwareDataType) System Version: macOS 10.12.2 (16C67) ========================================= $ pkgutil --pkg-info=com.apple.pkg.CLTools_Executables package-id: com.apple.pkg.CLTools_Executables ========================================= Apple LLVM version 8.0.0 (clang-800.0.42.1) InstalledDir:
|
@esd100 {} is for shell expansion (see 3.4.2 Brace expansion http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_03_04.html) 8.0lib was a typo, missed a '/', thanks for pointing that out. Fixed. @peterswang good point. The pre-compiled version doesn't have all the optimization flags. For best performance for a specific system, build from source. (ubuntu vs gentoo philosophy) The tensorflow guide on building from source is well documented: https://www.tensorflow.org/install/install_sources |
@esd100 Segmentation fault: 11 is usually followed by a OS X crash report. You can find crash reports in your Console application -> User Diagnostic Reports The segfault is usually a result of mixing your link libraries. You can avoid lots of these issues by using the docker images. |
@victorv Thanks. Sorry for the delay. This is the crash report after putting in the commands:
|
Have you tried setting I actually see similar Most likely thing is that there's a piece of code which isn't checking for null pointers, which can happen non-deterministically. For instance on MacOS, CFStringGetCStringPtr can return null non-deterministically. That output isn't checked for null and I fixed one such place in #3448 but I see other return values that aren't checked for null, such as output of |
@esd100 it could be LD_LIBRARY_PATH as @yaroslavvb has noted or you could be missing a cuda.1.dylib link in your /usr/local/cuda/lib directory. lrwxr-xr-x 1 root staff 33 Feb 16 12:33 libcuda.1.dylib -> /usr/local/cuda/lib/libcuda.dylib ln -s libcuda.dylib /usr/local/cuda/lib/libcuda.1.dylib |
@victorv @yaroslavvb I have the cuda.1.dylib link. Would you recommend setting LD_LIBRARY_PATH in the bash profile? What do you set it to? In my bash_profile, I have:
|
|
Thanks for the input commands. Some changes with that.
|
I added
and now ...
|
First trial run ...
Do you know if this output means everything is running appropriately? I'm not sure why there are all the warning about the tensorflow compiling without the special instruction sets. Any ideas? |
Everything is fine. The warnings just warn you that things could be faster. |
@martinwicke thanks! |
@esd100 you can suppress info and warning log messages in the shell with |
For me (MacBook Pro macOS Sierra + CUDA-8.0)
Then
It works... THX @yaroslavvb @esd100 |
After configuring and running
bazel build -c opt --config=cuda //tensorflow/tools/pip_package:build_pip_package
I am getting this error during the build:
ERROR: /pathtotensorflow/tensorflow/tensorflow/contrib/session_bundle/example/BUILD:38:1: Executing genrule //tensorflow/contrib/session_bundle/example:half_plus_two failed: bash failed: error executing command /bin/bash -c ... (remaining 1 argument(s) skipped): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1. Traceback (most recent call last): File "/private/var/tmp/_bazel_xxx/2ba3b3e08313f86d16eb1de7b54bf064/tensorflow/bazel-out/host/bin/tensorflow/contrib/session_bundle/example/export_half_plus_two.runfiles/tensorflow/contrib/session_bundle/example/export_half_plus_two.py", line 33, in <module> from tensorflow.contrib.session_bundle import exporter ImportError: No module named session_bundle Target //tensorflow/tools/pip_package:build_pip_package failed to build
Environment info
Operating System: os x el Capitan
Installed version of CUDA and cuDNN:
-rwxr-xr-x 1 root wheel 8280 Apr 12 23:02 /usr/local/cuda/lib/libcuda.dylib lrwxr-xr-x 1 root wheel 45 Apr 12 23:03 /usr/local/cuda/lib/libcudadevrt.a -> /Developer/NVIDIA/CUDA-7.5/lib/libcudadevrt.a lrwxr-xr-x 1 root wheel 50 Apr 12 23:03 /usr/local/cuda/lib/libcudart.7.5.dylib -> /Developer/NVIDIA/CUDA-7.5/lib/libcudart.7.5.dylib lrwxr-xr-x 1 root wheel 46 Apr 12 23:03 /usr/local/cuda/lib/libcudart.dylib -> /Developer/NVIDIA/CUDA-7.5/lib/libcudart.dylib lrwxr-xr-x 1 root wheel 49 Apr 12 23:03 /usr/local/cuda/lib/libcudart_static.a -> /Developer/NVIDIA/CUDA-7.5/lib/libcudart_static.a lrwxr-xr-x 1 root admin 47 Jun 16 11:25 /usr/local/cuda/lib/libcudnn.5.dylib -> /Developer/NVIDIA/CUDA-7.5/lib/libcudnn.5.dylib lrwxr-xr-x 1 root admin 45 Jun 16 11:25 /usr/local/cuda/lib/libcudnn.dylib -> /Developer/NVIDIA/CUDA-7.5/lib/libcudnn.dylib lrwxr-xr-x 1 root admin 48 Jun 16 11:25 /usr/local/cuda/lib/libcudnn_static.a -> /Developer/NVIDIA/CUDA-7.5/lib/libcudnn_static.a
The text was updated successfully, but these errors were encountered: