Installing Tensorflow with GPU support on os x 10.11 #2940

mts42000 · Jun 17, 2016

After configuring and running
bazel build -c opt --config=cuda //tensorflow/tools/pip_package:build_pip_package
I am getting this error during the build:

ERROR: /pathtotensorflow/tensorflow/tensorflow/contrib/session_bundle/example/BUILD:38:1: Executing genrule //tensorflow/contrib/session_bundle/example:half_plus_two failed: bash failed: error executing command /bin/bash -c ... (remaining 1 argument(s) skipped): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1. Traceback (most recent call last): File "/private/var/tmp/_bazel_xxx/2ba3b3e08313f86d16eb1de7b54bf064/tensorflow/bazel-out/host/bin/tensorflow/contrib/session_bundle/example/export_half_plus_two.runfiles/tensorflow/contrib/session_bundle/example/export_half_plus_two.py", line 33, in <module> from tensorflow.contrib.session_bundle import exporter ImportError: No module named session_bundle Target //tensorflow/tools/pip_package:build_pip_package failed to build

Environment info

Operating System: os x el Capitan

Installed version of CUDA and cuDNN:
-rwxr-xr-x 1 root wheel 8280 Apr 12 23:02 /usr/local/cuda/lib/libcuda.dylib lrwxr-xr-x 1 root wheel 45 Apr 12 23:03 /usr/local/cuda/lib/libcudadevrt.a -> /Developer/NVIDIA/CUDA-7.5/lib/libcudadevrt.a lrwxr-xr-x 1 root wheel 50 Apr 12 23:03 /usr/local/cuda/lib/libcudart.7.5.dylib -> /Developer/NVIDIA/CUDA-7.5/lib/libcudart.7.5.dylib lrwxr-xr-x 1 root wheel 46 Apr 12 23:03 /usr/local/cuda/lib/libcudart.dylib -> /Developer/NVIDIA/CUDA-7.5/lib/libcudart.dylib lrwxr-xr-x 1 root wheel 49 Apr 12 23:03 /usr/local/cuda/lib/libcudart_static.a -> /Developer/NVIDIA/CUDA-7.5/lib/libcudart_static.a lrwxr-xr-x 1 root admin 47 Jun 16 11:25 /usr/local/cuda/lib/libcudnn.5.dylib -> /Developer/NVIDIA/CUDA-7.5/lib/libcudnn.5.dylib lrwxr-xr-x 1 root admin 45 Jun 16 11:25 /usr/local/cuda/lib/libcudnn.dylib -> /Developer/NVIDIA/CUDA-7.5/lib/libcudnn.dylib lrwxr-xr-x 1 root admin 48 Jun 16 11:25 /usr/local/cuda/lib/libcudnn_static.a -> /Developer/NVIDIA/CUDA-7.5/lib/libcudnn_static.a

Velkan · Jun 21, 2016

Hey guys

I actually pass the bazel build -c opt --config=cuda //tensorflow/tools/pip_package:build_pip_package process.
But there is no wheel file in the bazel-bin/tensorflow/tools/pip_package/ and
the bazel-bin/tensorflow/tools/pip_package/build_pip_package is not a directory.

Anyone has any idear?
(I'm also on OS X 10.11, RMBP 2012 with GeForce GT 650M)

martinwicke · Jun 23, 2016

You have to run bazel-bin/tensorflow/tools/pip_package/build_pip_package.
See the installation instructions.
On Tue, Jun 21, 2016 at 06:28 Velkan.Xu notifications@github.com wrote:

Hey guys

I actually pass the bazel build -c opt --config=cuda
//tensorflow/tools/pip_package:build_pip_package process.
But there is no wheel file in the bazel-bin/tensorflow/tools/pip_package/
and
the bazel-bin/tensorflow/tools/pip_package/build_pip_package is not a
directory.

Anyone has any idear?

—
You are receiving this because you were assigned.

Reply to this email directly, view it on GitHub
#2940 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AAjO_Ym-8fCQzn1kP95IqZ1S5I7OZMl2ks5qN-b0gaJpZM4I4z4H
.

Velkan · Jun 23, 2016

Hi Martin:

Thanks to reply this, really.

I run 'bazel-bin/tensorflow/tools/pip_package/build_pip_package' but got a
error 'error: invalid command 'bdist_wheel''

I spent a day googled it and some said I need to install the wheel and
setuptools package which I already did. And this error still there.
Some said maybe there was a conflict between the old version of setuptools
and pip, but setuptools and pip on my macbook were both updated. And the error
still there.

There is a closed issue: #348
This guy andyyuan78 https://github.com/andyyuan78 said that he got this
error before and solved it by run 'pip2 install wheel'.
I tried both 'pip install' and 'pip3 install' but it didn't work.
I'm using Python3.5 and this is the only Python I have on my macbook.

Any ideas?

Martin Wicke notifications@github.com于2016年6月23日周四上午11:32写道：

You have to run bazel-bin/tensorflow/tools/pip_package/build_pip_package.
See the installation instructions.
On Tue, Jun 21, 2016 at 06:28 Velkan.Xu notifications@github.com wrote:

Hey guys

I actually pass the bazel build -c opt --config=cuda
//tensorflow/tools/pip_package:build_pip_package process.
But there is no wheel file in the bazel-bin/tensorflow/tools/pip_package/
and
the bazel-bin/tensorflow/tools/pip_package/build_pip_package is not a
directory.

Anyone has any idear?

—
You are receiving this because you were assigned.

Reply to this email directly, view it on GitHub
<
#2940 (comment)
,
or mute the thread
<
https://github.com/notifications/unsubscribe/AAjO_Ym-8fCQzn1kP95IqZ1S5I7OZMl2ks5qN-b0gaJpZM4I4z4H

.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#2940 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/ADmW7B6IUAM_ftseeymPcaSblzZs3PIMks5qOf5IgaJpZM4I4z4H
.

peterswang · Jun 23, 2016

I ran into a similar issue with Python 2.7.11 on OS X 10.11.5 + MBP.

$ ./configure
Please specify the location of python. [Default is /usr/local/bin/python]:
Do you wish to build TensorFlow with Google Cloud Platform support? [y/N] N
No Google Cloud Platform support will be enabled for TensorFlow
Do you wish to build TensorFlow with GPU support? [y/N] y
GPU support will be enabled for TensorFlow
Please specify which gcc nvcc should use as the host compiler. [Default is /usr/bin/gcc]:
Please specify the Cuda SDK version you want to use, e.g. 7.0. [Leave empty to use system default]: 7.5
Please specify the location where CUDA 7.5 toolkit is installed. Refer to README.md for more details. [Default is /usr/local/cuda]:
Please specify the Cudnn version you want to use. [Leave empty to use system default]: 5
Please specify the location where cuDNN 5 library is installed. Refer to README.md for more details. [Default is /usr/local/cuda]:
Please specify a list of comma-separated Cuda compute capabilities you want to build with.
You can find the compute capability of your device at: https://developer.nvidia.com/cuda-gpus.
Please note that each additional compute capability significantly increases your build time and binary size.

Setting up Cuda include
Setting up Cuda lib
Setting up Cuda bin
Setting up Cuda nvvm
Setting up CUPTI include
Setting up CUPTI lib64
Configuration finished
$ bazel build -c opt --config=cuda //tensorflow/tools/pip_package:build_pip_package
.......
.......
1 warning generated.
INFO: From Linking tensorflow/python/_pywrap_tensorflow.so [for host]:
clang: warning: argument unused during compilation: '-pthread'
ld: warning: option -noall_load is obsolete and being ignored
INFO: From Linking tensorflow/python/_pywrap_tensorflow.so:
clang: warning: argument unused during compilation: '-pthread'
ld: warning: option -noall_load is obsolete and being ignored
ERROR: /Users/peter_wang/tensorflow/tensorflow/contrib/session_bundle/example/BUILD:38:1: Executing genrule //tensorflow/contrib/session_bundle/example:half_plus_two failed: bash failed: error executing command /bin/bash -c ... (remaining 1 argument(s) skipped): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 245.
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcublas.7.5.dylib locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcudnn.5.dylib locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcufft.7.5.dylib locally
Target //tensorflow/tools/pip_package:build_pip_package failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 1826.337s, Critical Path: 1821.08s

To verify CUDA install:
$ ./bin/x86_64/darwin/release/deviceQuery
./bin/x86_64/darwin/release/deviceQuery Starting...

CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 1 CUDA Capable device(s)

Device 0: "GeForce GT 650M"
CUDA Driver Version / Runtime Version 7.5 / 7.5
CUDA Capability Major/Minor version number: 3.0
Total amount of global memory: 512 MBytes (536543232 bytes)
( 2) Multiprocessors, (192) CUDA Cores/MP: 384 CUDA Cores
GPU Max Clock rate: 405 MHz (0.41 GHz)
Memory Clock rate: 2000 Mhz
Memory Bus Width: 128-bit
L2 Cache Size: 262144 bytes
Maximum Texture Dimension Size (x,y,z) 1D=(65536), 2D=(65536, 65536), 3D=(4096, 4096, 4096)
Maximum Layered 1D Texture Size, (num) layers 1D=(16384), 2048 layers
Maximum Layered 2D Texture Size, (num) layers 2D=(16384, 16384), 2048 layers
Total amount of constant memory: 65536 bytes
Total amount of shared memory per block: 49152 bytes
Total number of registers available per block: 65536
Warp size: 32
Maximum number of threads per multiprocessor: 2048
Maximum number of threads per block: 1024
Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
Max dimension size of a grid size (x,y,z): (2147483647, 65535, 65535)
Maximum memory pitch: 2147483647 bytes
Texture alignment: 512 bytes
Concurrent copy and kernel execution: Yes with 1 copy engine(s)
Run time limit on kernels: Yes
Integrated GPU sharing Host Memory: No
Support host page-locked memory mapping: Yes
Alignment requirement for Surfaces: Yes
Device has ECC support: Disabled
Device supports Unified Addressing (UVA): Yes
Device PCI Domain ID / Bus ID / location ID: 0 / 1 / 0
Compute Mode:
< Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 7.5, CUDA Runtime Version = 7.5, NumDevs = 1, Device0 = GeForce GT 650M
Result = PASS

Velkan · Jun 24, 2016

Hi peter:

I actually pass the build process.

Maybe you want set a cuda capability during the tensorflow configuration?
When it like:
'Please specify a list of comma-separated Cuda compute capabilities you
want to build with.'
Enter '3.0' for the gt 650m.

And your error log said:
'Use --verbose_failures to see the command lines of failed build steps.'
Maybe you can get some details by doing that?

And make sure your source code is updated. I got a wired error once by
using a old version of the source and solved it by clone from the master.

On Fri, Jun 24, 2016, 3:04 AM peterswang notifications@github.com wrote:

I ran into a similar issue with Python 2.7.11 on OS X 10.11.5 + MBP.

$ ./configure
Please specify the location of python. [Default is /usr/local/bin/python]:
Do you wish to build TensorFlow with Google Cloud Platform support? [y/N] N
No Google Cloud Platform support will be enabled for TensorFlow
Do you wish to build TensorFlow with GPU support? [y/N] y
GPU support will be enabled for TensorFlow
Please specify which gcc nvcc should use as the host compiler. [Default is
/usr/bin/gcc]:
Please specify the Cuda SDK version you want to use, e.g. 7.0. [Leave
empty to use system default]: 7.5
Please specify the location where CUDA 7.5 toolkit is installed. Refer to
README.md for more details. [Default is /usr/local/cuda]:
Please specify the Cudnn version you want to use. [Leave empty to use
system default]: 5
Please specify the location where cuDNN 5 library is installed. Refer to
README.md for more details. [Default is /usr/local/cuda]:
Please specify a list of comma-separated Cuda compute capabilities you
want to build with.
You can find the compute capability of your device at:
https://developer.nvidia.com/cuda-gpus.
Please note that each additional compute capability significantly
increases your build time and binary size.

Setting up Cuda include
Setting up Cuda lib
Setting up Cuda bin
Setting up Cuda nvvm
Setting up CUPTI include
Setting up CUPTI lib64
Configuration finished
$ bazel build -c opt --config=cuda
//tensorflow/tools/pip_package:build_pip_package
.......
.......
1 warning generated.
INFO: From Linking tensorflow/python/_pywrap_tensorflow.so [for host]:
clang: warning: argument unused during compilation: '-pthread'
ld: warning: option -noall_load is obsolete and being ignored
INFO: From Linking tensorflow/python/_pywrap_tensorflow.so:
clang: warning: argument unused during compilation: '-pthread'
ld: warning: option -noall_load is obsolete and being ignored
ERROR:
/Users/peter_wang/tensorflow/tensorflow/contrib/session_bundle/example/BUILD:38:1:
Executing genrule //tensorflow/contrib/session_bundle/example:half_plus_two
failed: bash failed: error executing command /bin/bash -c ... (remaining 1
argument(s) skipped):
com.google.devtools.build.lib.shell.BadExitStatusException: Process exited
with status 245.
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA
library libcublas.7.5.dylib locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA
library libcudnn.5.dylib locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA
library libcufft.7.5.dylib locally
Target //tensorflow/tools/pip_package:build_pip_package failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 1826.337s, Critical Path: 1821.08s

To verify CUDA install:
$ ./bin/x86_64/darwin/release/deviceQuery
./bin/x86_64/darwin/release/deviceQuery Starting...

CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 1 CUDA Capable device(s)

Device 0: "GeForce GT 650M"
CUDA Driver Version / Runtime Version 7.5 / 7.5
CUDA Capability Major/Minor version number: 3.0
Total amount of global memory: 512 MBytes (536543232 bytes)
( 2) Multiprocessors, (192) CUDA Cores/MP: 384 CUDA Cores
GPU Max Clock rate: 405 MHz (0.41 GHz)
Memory Clock rate: 2000 Mhz
Memory Bus Width: 128-bit
L2 Cache Size: 262144 bytes
Maximum Texture Dimension Size (x,y,z) 1D=(65536), 2D=(65536, 65536),
3D=(4096, 4096, 4096)
Maximum Layered 1D Texture Size, (num) layers 1D=(16384), 2048 layers
Maximum Layered 2D Texture Size, (num) layers 2D=(16384, 16384), 2048
layers
Total amount of constant memory: 65536 bytes
Total amount of shared memory per block: 49152 bytes
Total number of registers available per block: 65536
Warp size: 32
Maximum number of threads per multiprocessor: 2048
Maximum number of threads per block: 1024
Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
Max dimension size of a grid size (x,y,z): (2147483647, 65535, 65535)
Maximum memory pitch: 2147483647 bytes
Texture alignment: 512 bytes
Concurrent copy and kernel execution: Yes with 1 copy engine(s)
Run time limit on kernels: Yes
Integrated GPU sharing Host Memory: No
Support host page-locked memory mapping: Yes
Alignment requirement for Surfaces: Yes
Device has ECC support: Disabled
Device supports Unified Addressing (UVA): Yes
Device PCI Domain ID / Bus ID / location ID: 0 / 1 / 0
Compute Mode:
< Default (multiple host threads can use ::cudaSetDevice() with device
simultaneously) >

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 7.5, CUDA Runtime
Version = 7.5, NumDevs = 1, Device0 = GeForce GT 650M
Result = PASS

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#2940 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/ADmW7AoIow1wLqEj_Jk1rli5ifANu3Zlks5qOtjUgaJpZM4I4z4H
.

peterswang · Jun 24, 2016

Thx for your suggestions, Velkan. I followed the instructions under the Build from Source section of the TF [Download and Setup] page. Compute capability was specified to be '3.0', just that the copy-paste missed that line.

The Github TF page shows last successful build for Mac OS to be 14 days ago. I just tried building again with a fresh clone from the repo and it crashed at the same place. Actually, bazel apparently invoked Python, which crashed, seemingly while trying to access the CUDA library, perhaps for libcufftw.7.5.dylib?

If it may be of any help to Martin, below is part of call stack from the crash report:
...
Crashed Thread: 0 Dispatch queue: com.apple.main-thread

Exception Type: EXC_BAD_ACCESS (SIGSEGV)
Exception Codes: KERN_INVALID_ADDRESS at 0x0000000000000000

VM Regions Near 0:
-->
__TEXT 000000010aec3000-000000010aec5000 [ 8K] r-x/rwx SM=COW /usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0 libsystem_c.dylib 0x00007fff8e610152 strlen + 18
1 _pywrap_tensorflow.so 0x000000010d3e19a2 perftools::gputools::internal::DsoLoader::GetDsoHandle(tensorflow::StringPiece, void**, perftools::gputools::internal::DsoLoader::LoadKind) + 402
2 _pywrap_tensorflow.so 0x000000010d3e2692 perftools::gputools::internal::DsoLoader::GetLibcudaDsoHandle(void**) + 258
3 _pywrap_tensorflow.so 0x000000010d0d1f22 std::__1::__function::__func<tensorflow::Status ()(tensorflow::shape_inference::InferenceContext), std::1::allocator<tensorflow::Status ()(tensorflow::shape_inference::InferenceContext)>, tensorflow::Status (tensorflow::shape_inference::InferenceContext)>::operator()(tensorflow::shape_inference::InferenceContext&&) + 18
4 _pywrap_tensorflow.so 0x000000010d3e2f7f perftools::gputools::internal::CachedDsoLoader::FetchHandleResult(std::__1::function<tensorflow::Status (void**)>) + 47
5 _pywrap_tensorflow.so 0x000000010d3e34e0 perftools::gputools::internal::CachedDsoLoader::GetLibcudaDsoHandle() + 112
6 _pywrap_tensorflow.so 0x000000010d464fe5 perftools::gputools::initialize_cuda_gpu_executor() + 37
7 dyld 0x00007fff60b5a10b ImageLoaderMachO::doModInitFunctions(ImageLoader::LinkContext const&) + 265
8 dyld 0x00007fff60b5a284 ImageLoaderMachO::doInitialization(ImageLoader::LinkContext const&) + 40
9 dyld 0x00007fff60b568bd ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 305
10 dyld 0x00007fff60b56743 ImageLoader::processInitializers(ImageLoader::LinkContext const&, unsigned int, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 127
11 dyld 0x00007fff60b569b3 ImageLoader::runInitializers(ImageLoader::LinkContext const&, ImageLoader::InitializerTimingList&) + 75
12 dyld 0x00007fff60b4beb0 dyld::runInitializers(ImageLoader*) + 89
13 dyld 0x00007fff60b53308 dlopen + 555
14 libdyld.dylib 0x00007fff8d7ae79c dlopen + 59
15 org.python.python 0x000000010af807d9 _PyImport_GetDynLoadFunc + 309
...

tomnielsen · Jun 29, 2016

Just another datapoint. I'm getting the same python crash using
bazel build -c opt --config=cuda //tensorflow/tools/pip_package:build_pip_package

After Python crashes for bazel build near the end, then
bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg
gives the error
error: invalid command 'bdist_wheel'
(Note: the directory /tmp/tensorflow_pkg does not exist. Probably because of the python crash)

Possible useful config information:

System:
  System Version:   OS X 10.10.5 (14F1808)
  Kernel Version:   Darwin 14.5.0
  Chipset Model:    NVIDIA GeForce GTX 760
  Type: GPU
  Bus:  PCIe
  PCIe Lane Width:  x16
  VRAM (Total): 4096 MB
  Vendor:   NVIDIA (0x10de)
  Device ID:    0x1187
  Revision ID:  0x00a2
  ROM Revision: preset 1.0.0


> python --version
Python 2.7.11

> /Developer/NVIDIA/CUDA-7.5/bin/nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2015 NVIDIA Corporation
Built on Thu_Sep_24_00:26:39_CDT_2015
Cuda compilation tools, release 7.5, V7.5.19

>pip --version
pip 8.1.2 from /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages (python 2.7)

> bazel version
Build label: 0.2.3-homebrew
Build target: bazel-out/local-fastbuild/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Tue May 17 15:07:33 2016 (1463497653)
Build timestamp: 1463497653
Build timestamp as int: 1463497653

Crash stack

Crashed Thread:        0  Dispatch queue: com.apple.main-thread

Exception Type:        EXC_BAD_ACCESS (SIGSEGV)
Exception Codes:       KERN_INVALID_ADDRESS at 0x0000000000000000

VM Regions Near 0:
--> 
    __TEXT                 0000000100000000-0000000100001000 [    4K] r-x/rwx SM=COW  /Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   libsystem_c.dylib               0x00007fff9666cf92 strlen + 18
1   _pywrap_tensorflow.so           0x00000001058ff490 perftools::gputools::internal::DsoLoader::GetDsoHandle(tensorflow::StringPiece, void**, perftools::gputools::internal::DsoLoader::LoadKind) + 384
2   _pywrap_tensorflow.so           0x000000010590000c perftools::gputools::internal::DsoLoader::GetLibcudaDsoHandle(void**) + 188
3   _pywrap_tensorflow.so           0x0000000105901072 std::__1::__function::__func<tensorflow::Status (*)(void**), std::__1::allocator<tensorflow::Status (*)(void**)>, tensorflow::Status (void**)>::operator()(void**&&) + 18
4   _pywrap_tensorflow.so           0x000000010590080f perftools::gputools::internal::CachedDsoLoader::FetchHandleResult(std::__1::function<tensorflow::Status (void**)>) + 47
5   _pywrap_tensorflow.so           0x0000000105900d6f perftools::gputools::internal::CachedDsoLoader::GetLibcudaDsoHandle() + 111
6   _pywrap_tensorflow.so           0x0000000105982b85 perftools::gputools::initialize_cuda_gpu_executor() + 37
7   dyld                            0x00007fff63346d4b ImageLoaderMachO::doModInitFunctions(ImageLoader::LinkContext const&) + 265
8   dyld                            0x00007fff63346ed8 ImageLoaderMachO::doInitialization(ImageLoader::LinkContext const&) + 40
9   dyld                            0x00007fff633438d1 ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 305
10  dyld                            0x00007fff63343758 ImageLoader::processInitializers(ImageLoader::LinkContext const&, unsigned int, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 138
11  dyld                            0x00007fff633439c9 ImageLoader::runInitializers(ImageLoader::LinkContext const&, ImageLoader::InitializerTimingList&) + 75
12  dyld                            0x00007fff633390ca dyld::runInitializers(ImageLoader*) + 89
13  dyld                            0x00007fff63340231 dlopen + 578
14  libdyld.dylib                   0x00007fff9005c857 dlopen + 59
15  org.python.python               0x00000001000fc86f _PyImport_GetDynLoadFunc + 303
...

yaroslavvb · Jun 30, 2016

@peterswang btw, I have seen see this strlen + 18 crash when

DSO was is not in the library path
I upgraded to El Capitan which enabled System Integrity Protection (this affects how DSOs get loaded)

martinwicke · Jun 30, 2016

There's a bunch of separate issues on this thread. The half_plus_two issue should have been resolved since. @tomnielsen @Velkan as for the wheel problem -- I haven't seen that other than when there's a lingering old version of setuptools around. Can you try installing in a virtualenv? Especially on Mac that often helps.

Invisibility · Jul 3, 2016

I also hit the same issue on Mac. I investigated a little bit.

As mentioned earlier in this thread, the crash happens because tensorflow couldn't load a cuda shared library. tensorflow shouldn't crash but it did because LOG(INFO) << ... << getenv("LD_LIBRARY_PATH") doesn't support char pointer being nullptr. I enclosed the relevant code at the end of this message.

As for why tensorflow didn't find the cuda shared library, in my case, it was because ipython from anaconda doesn't inherit the environment variable from my bash shell. I switched to use python instead of ipython, tensorflow was able to load the cuda shared library.

Hopefully this finding will help others who have similar problem.

tensorflow/stream_executor/dso_loader.cc@93

/* static */ port::Status DsoLoader::GetDsoHandle(port::StringPiece path,
                                                  void** dso_handle,
                                                  LoadKind load_kind) {

  int dynload_flags =
      RTLD_LAZY | (load_kind == LoadKind::kLocal ? RTLD_LOCAL : RTLD_GLOBAL);
  string path_string = path.ToString();
  *dso_handle = dlopen(path_string.c_str(), dynload_flags);
  if (*dso_handle == nullptr) {
    LOG(INFO) << "Couldn't open CUDA library " << path
              << ". LD_LIBRARY_PATH: " << getenv("LD_LIBRARY_PATH");
    return port::Status(
        port::error::FAILED_PRECONDITION,
        port::StrCat("could not dlopen DSO: ", path, "; dlerror: ", dlerror()));
  }
  LOG(INFO) << "successfully opened CUDA library " << path
            << (load_kind == LoadKind::kLocal ? " locally" : " globally");
  return port::Status::OK();
}

probably common.cc in protobuf

LogMessage& LogMessage::operator<<(const char* value) {
  message_ += value;
  return *this;
}

XCode c++ library string

...

_LIBCPP_INLINE_VISIBILITY basic_string& operator+=(const value_type* __s)         {return append(__s);}

...

basic_string<_CharT, _Traits, _Allocator>::append(const value_type* __s)
{
    _LIBCPP_ASSERT(__s != nullptr, "string::append received nullptr");
    return append(__s, traits_type::length(__s));
}

...

static inline size_t length(const char_type* __s) {return strlen(__s);}

yaroslavvb · Jul 3, 2016

@Invisibility nice digging. I've seen similar looking crashes on Mac in other places with similar stack trace, now I know where to look for. BTW, you can set env var in anaconda directly using something like this

import os
os.environ["LD_LIBRARY_PATH"]=....
import tensorflow

Invisibility · Jul 4, 2016

Thanks for the tip, @yaroslavvb. I tried, but it still crashed. It seems that for some reason dlopen in DsoLoader::GetDsoHandle didn't see the environment variable. I noticed that in my environment, ipython is a script with the following content:

#!/bin/bash <my anaconda virtual environment path>/bin/python.app
if __name__ == '__main__':
    import sys
    import IPython

    sys.exit(IPython.start_ipython())

My workaround is to avoid using python.app, i.e. running the following command insteead of directly running ipython:

python <my anaconda virtual environment path>/bin/ipython

hbfs · Jul 26, 2016

I'm also getting a segmentation fault in python.

$ python test.py
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcublas.7.5.dylib locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcudnn.5.dylib locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcufft.7.5.dylib locally
[1]    34129 segmentation fault  python test.py

test.py

import tensorflow as tf

# Creates a graph.
a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3], name='a')
b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2], name='b')
c = tf.matmul(a, b)

# Creates a session with log_device_placement set to True.
sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))

# Runs the op.
print sess.run(c)

System Info:

Process:               Python [34031]
Path:                  /usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python
Identifier:            Python
Version:               2.7.11 (2.7.11)
Code Type:             X86-64 (Native)
Parent Process:        zsh [33999]
Responsible:           iTerm2 [508]

OS Version:            Mac OS X 10.11.6 (15G31)
Kernel Version: Darwin 15.6.0
Graphics: NVIDIA GeForce GTX 960, NVIDIA GeForce GTX 960, PCIe

Stack Trace

Crashed Thread:        0  Dispatch queue: com.apple.main-thread

Exception Type:        EXC_BAD_ACCESS (SIGSEGV)
Exception Codes:       KERN_INVALID_ADDRESS at 0x0000000000000000

VM Regions Near 0:
--> 
    __TEXT                 000000010782c000-000000010782e000 [    8K] r-x/rwx SM=COW  /usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   libsystem_c.dylib               0x00007fff8ce51132 strlen + 18
1   _pywrap_tensorflow.so           0x000000010a02b912 perftools::gputools::internal::DsoLoader::GetDsoHandle(tensorflow::StringPiece, void**, perftools::gputools::internal::DsoLoader::LoadKind) + 402
2   _pywrap_tensorflow.so           0x000000010a02c602 perftools::gputools::internal::DsoLoader::GetLibcudaDsoHandle(void**) + 258
3   _pywrap_tensorflow.so           0x0000000109e44002 std::__1::__function::__func<tensorflow::Status (*)(tensorflow::shape_inference::InferenceContext*), std::__1::allocator<tensorflow::Status (*)(tensorflow::shape_inference::InferenceContext*)>, tensorflow::Status (tensorflow::shape_inference::InferenceContext*)>::operator()(tensorflow::shape_inference::InferenceContext*&&) + 18
4   _pywrap_tensorflow.so           0x000000010a02ceef perftools::gputools::internal::CachedDsoLoader::FetchHandleResult(std::__1::function<tensorflow::Status (void**)>) + 47
5   _pywrap_tensorflow.so           0x000000010a02d450 perftools::gputools::internal::CachedDsoLoader::GetLibcudaDsoHandle() + 112
6   _pywrap_tensorflow.so           0x000000010a0b0195 perftools::gputools::initialize_cuda_gpu_executor() + 37
7   dyld                            0x00007fff6cb8710b ImageLoaderMachO::doModInitFunctions(ImageLoader::LinkContext const&) + 265
8   dyld                            0x00007fff6cb87284 ImageLoaderMachO::doInitialization(ImageLoader::LinkContext const&) + 40
9   dyld                            0x00007fff6cb838bd ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 305
10  dyld                            0x00007fff6cb83743 ImageLoader::processInitializers(ImageLoader::LinkContext const&, unsigned int, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 127
11  dyld                            0x00007fff6cb839b3 ImageLoader::runInitializers(ImageLoader::LinkContext const&, ImageLoader::InitializerTimingList&) + 75
12  dyld                            0x00007fff6cb78eb0 dyld::runInitializers(ImageLoader*) + 89
13  dyld                            0x00007fff6cb80308 dlopen + 555
14  libdyld.dylib                   0x00007fff960d779c dlopen + 59
15  org.python.python               0x00000001078e7892 _PyImport_GetDynLoadFunc + 309
...

edit: The issue occurs because the environment variable LD_LIBRARY_PATH is not set. I set it to the same as DYLD_LIBRARY_PATH which resolved the segmentation fault from a null path as discovered above in #2940 (comment) by @Invisibility .

I then encountered this error:

I tensorflow/stream_executor/dso_loader.cc:102] Couldn't open CUDA library libcuda.1.dylib.

I created a symlink to in /usr/local/cuda/lib with ln -s libcuda.dylib libcuda.1.dylib and now I'm getting a different segmentation fault:

Crashed Thread:        0  Dispatch queue: com.apple.main-thread

Exception Type:        EXC_BAD_ACCESS (SIGSEGV)
Exception Codes:       EXC_I386_GPFLT

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   _pywrap_tensorflow.so           0x000000010bb73e0b perftools::gputools::StreamExecutor::DeviceMemoryUsage(long long*, long long*) const + 11
1   _pywrap_tensorflow.so           0x000000010b9587fe tensorflow::GPUMachineManager() + 382
2   _pywrap_tensorflow.so           0x000000010b9564ce tensorflow::BaseGPUDeviceFactory::GetValidDeviceIds(std::__1::vector<int, std::__1::allocator<int> >*) + 46
3   _pywrap_tensorflow.so           0x000000010b9562f9 tensorflow::BaseGPUDeviceFactory::CreateDevices(tensorflow::SessionOptions const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::vector<tensorflow::Device*, std::__1::allocator<tensorflow::Device*> >*) + 345
4   _pywrap_tensorflow.so           0x000000010bae3795 tensorflow::DeviceFactory::AddDevices(tensorflow::SessionOptions const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::vector<tensorflow::Device*, std::__1::allocator<tensorflow::Device*> >*) + 245
5   _pywrap_tensorflow.so           0x000000010b912c15 tensorflow::DirectSessionFactory::NewSession(tensorflow::SessionOptions const&) + 133
6   _pywrap_tensorflow.so           0x000000010bb096c8 tensorflow::NewSession(tensorflow::SessionOptions const&, tensorflow::Session**) + 184
7   _pywrap_tensorflow.so           0x000000010bace8d1 TF_NewSession + 33
8   _pywrap_tensorflow.so           0x000000010a4845b2 _wrap_TF_NewSession(_object*, _object*) + 162
9   org.python.python               0x0000000109424f1b PyEval_EvalFrameEx + 27072
...

bh4cyi · Jul 26, 2016

Try Python3 instead of Python2

hbfs · Jul 26, 2016

I'm currently using a GTX 960 2GB at 4096x2160@60Hz. Based on #2840, I wonder if all the VRAM is used up for the display which is why I'm getting a segmentation fault? Also Chrome is using 49.18 GB / 64 GB system memory..

I have a GTX 960 4GB that I can install and test tomorrow.

hbfs · Jul 26, 2016

Previously I was following these instructions https://gist.github.com/Mistobaan/dd32287eeb6859c6668d and built from master. Now I used https://gist.github.com/ageitgey/819a51afa4613649bd18 and built from tag cuda_osx and it worked. Note: I had to run git submodule update --init prior to bazel build commands and also prior to creating the pip packages cp -r bazel-bin/tensorflow/tools/pip_package/build_pip_package.runfiles/__main__/* bazel-bin/tensorflow/tools/pip_package/build_pip_package.runfiles/ from #2040 (comment) Notice the amount of free memory available!

python test.py
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcublas.7.5.dylib locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcudnn.5.dylib locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcufft.7.5.dylib locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcuda.1.dylib locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcurand.7.5.dylib locally
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:892] OS X does not support NUMA - returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 0 with properties:
name: GeForce GTX 960
major: 5 minor: 2 memoryClockRate (GHz) 1.367
pciBusID 0000:04:00.0
Total memory: 2.00GiB
Free memory: 176.32MiB
I tensorflow/core/common_runtime/gpu/gpu_init.cc:126] DMA: 0
I tensorflow/core/common_runtime/gpu/gpu_init.cc:136] 0:   Y
I tensorflow/core/common_runtime/gpu/gpu_device.cc:841] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 960, pci bus id: 0000:04:00.0)
Device mapping:
/job:localhost/replica:0/task:0/gpu:0 -> device: 0, name: GeForce GTX 960, pci bus id: 0000:04:00.0
I tensorflow/core/common_runtime/direct_session.cc:175] Device mapping:
/job:localhost/replica:0/task:0/gpu:0 -> device: 0, name: GeForce GTX 960, pci bus id: 0000:04:00.0

MatMul: /job:localhost/replica:0/task:0/gpu:0
I tensorflow/core/common_runtime/simple_placer.cc:818] MatMul: /job:localhost/replica:0/task:0/gpu:0
b: /job:localhost/replica:0/task:0/gpu:0
I tensorflow/core/common_runtime/simple_placer.cc:818] b: /job:localhost/replica:0/task:0/gpu:0
a: /job:localhost/replica:0/task:0/gpu:0
I tensorflow/core/common_runtime/simple_placer.cc:818] a: /job:localhost/replica:0/task:0/gpu:0
[[ 22.  28.]
 [ 49.  64.]]

I also rebuilt this using master branch and it worked. I think the key is TF_UNOFFICIAL_SETTING=1 ./configure

yaroslavvb · Jul 26, 2016

BTW, there's now a daily wheel with Mac GPU support:

export url=http://ci.tensorflow.org/view/Nightly/job/nigntly-matrix-mac-gpu/TF_BUILD_IS_OPT=OPT,TF_BUILD_IS_PIP=PIP,TF_BUILD_PYTHON_VERSION=PYTHON3,label=gpu-mac/lastSuccessfulBuild/artifact/pip_test/whl/tensorflow-0.9.0-py3-none-any.whl
pip install --upgrade $url

alexcolburn · Aug 10, 2016

There is a bug with loading libcuda.dylib - the default cuda install creates libcuda.dylib, but tensorflow tries to load libcuda.1.dylib . This fails, resorting to using the LD_LIBRARY_PATH which if NULL crashes. If you copy libcuda.dylib to libcuda.1.dylib it loads fine.

xuanchien · Sep 2, 2016

Thanks @alexcolburn, copying libcuda.dylib to libcuda.1.dylib indeed fixed this issue on my Mac 10.10

vinhqdang · Sep 28, 2016

I confirmed comment of @xuanchien . So a simple solution. Worked on my MBP 2014 (nvidia 750M) with Mac OS 10.12

leocnj · Sep 28, 2016

Thanks. I applied the same trick and now my TF on python 2.7 on OSX 10.11.5 works.

martinwicke · Nov 29, 2016

I'm closing this issue, I believe we have fixed the cuda version problem as well.

olegsinyavskiy · Jan 15, 2017

The issues still persist in tensorflow 1.0.0 alpha:

python -c "import tensorflow; print(tensorflow.__version__)"
I tensorflow/stream_executor/dso_loader.cc:125] successfully opened CUDA library libcublas.8.0.dylib locally
I tensorflow/stream_executor/dso_loader.cc:125] successfully opened CUDA library libcudnn.5.dylib locally
I tensorflow/stream_executor/dso_loader.cc:125] successfully opened CUDA library libcufft.8.0.dylib locally
I tensorflow/stream_executor/dso_loader.cc:116] Couldn't open CUDA library libcuda.1.dylib. LD_LIBRARY_PATH: :/usr/local/cuda/lib
I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:165] hostname: Olegs-MacBook-Pro-4.local
I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:189] libcuda reported version is: Not found: was unable to find libcuda.so DSO loaded into this program
I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:193] kernel reported version is: Invalid argument: expected %d.%d or %d.%d.%d form for driver version; got ""
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1065] LD_LIBRARY_PATH: :/usr/local/cuda/lib
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1066] failed to find libcuda.so on this system: Failed precondition: could not dlopen DSO: libcuda.1.dylib; dlerror: dlopen(libcuda.1.dylib, 6): image not found
I tensorflow/stream_executor/dso_loader.cc:125] successfully opened CUDA library libcurand.8.0.dylib locally
1.0.0-alpha

alexcolburn advice to link libcuda.1.dylib to libcuda.dylib does help.

kylemcdonald · Feb 16, 2017

Same problem after installing Tensorflow today. Version info:

$ python --version
Python 2.7.13
$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2016 NVIDIA Corporation
Built on Sun_Oct_30_22:18:43_CDT_2016
Cuda compilation tools, release 8.0, V8.0.54
$ gcc --version
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 8.0.0 (clang-800.0.42.1)
Target: x86_64-apple-darwin15.6.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

Problem description:

$ python -c "import tensorflow"
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcublas.8.0.dylib locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcudnn.5.dylib locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcufft.8.0.dylib locally
I tensorflow/stream_executor/dso_loader.cc:126] Couldn't open CUDA library libcuda.1.dylib. LD_LIBRARY_PATH: /Developer/NVIDIA/CUDA-8.0/lib
I tensorflow/stream_executor/dso_loader.cc:126] Couldn't open CUDA library libcuda.dylib. LD_LIBRARY_PATH: /Developer/NVIDIA/CUDA-8.0/lib
...

After creating a symlink:

$ sudo ln -s /usr/local/cuda/lib/libcuda.dylib /Developer/NVIDIA/CUDA-8.0/lib/libcuda.1.dylib
$ python -c "import tensorflow"
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcublas.8.0.dylib locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcudnn.5.dylib locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcufft.8.0.dylib locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcuda.1.dylib locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcurand.8.0.dylib locally
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python2.7/site-packages/tensorflow/__init__.py", line 24, in <module>
    from tensorflow.python import *
  File "/usr/local/lib/python2.7/site-packages/tensorflow/python/__init__.py", line 75, in <module>
    from tensorflow.core.framework.graph_pb2 import *
  File "/usr/local/lib/python2.7/site-packages/tensorflow/core/framework/graph_pb2.py", line 16, in <module>
    from tensorflow.core.framework import node_def_pb2 as tensorflow_dot_core_dot_framework_dot_node__def__pb2
  File "/usr/local/lib/python2.7/site-packages/tensorflow/core/framework/node_def_pb2.py", line 16, in <module>
    from tensorflow.core.framework import attr_value_pb2 as tensorflow_dot_core_dot_framework_dot_attr__value__pb2
  File "/usr/local/lib/python2.7/site-packages/tensorflow/core/framework/attr_value_pb2.py", line 16, in <module>
    from tensorflow.core.framework import tensor_pb2 as tensorflow_dot_core_dot_framework_dot_tensor__pb2
  File "/usr/local/lib/python2.7/site-packages/tensorflow/core/framework/tensor_pb2.py", line 16, in <module>
    from tensorflow.core.framework import resource_handle_pb2 as tensorflow_dot_core_dot_framework_dot_resource__handle__pb2
  File "/usr/local/lib/python2.7/site-packages/tensorflow/core/framework/resource_handle_pb2.py", line 22, in <module>
    serialized_pb=_b('\n/tensorflow/core/framework/resource_handle.proto\x12\ntensorflow\"m\n\x0eResourceHandle\x12\x0e\n\x06\x64\x65vice\x18\x01 \x01(\t\x12\x11\n\tcontainer\x18\x02 \x01(\t\x12\x0c\n\x04name\x18\x03 \x01(\t\x12\x11\n\thash_code\x18\x04 \x01(\x04\x12\x17\n\x0fmaybe_type_name\x18\x05 \x01(\tB4\n\x18org.tensorflow.frameworkB\x13ResourceHandleProtoP\x01\xf8\x01\x01\x62\x06proto3')
TypeError: __init__() got an unexpected keyword argument 'syntax'

The remaining error was due to protobuf not being installed correctly. I had previously run brew uninstall protobuf. Once I uninstalled it, Tensorflow works correctly:

$ brew uninstall protobuf
$ python -c "import tensorflow"
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcublas.8.0.dylib locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcudnn.5.dylib locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcufft.8.0.dylib locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcuda.1.dylib locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcurand.8.0.dylib locally
$

victorv · Feb 16, 2017

This is still a common problem on OS X 10.11.5 / 10.11.6 as we have new team members following the instructions at https://www.tensorflow.org/install/install_mac

The solution is to create a link in /usr/local/cuda/lib with 'sudo ln -s /usr/local/cuda/lib/libcuda.dylib /usr/local/cuda/lib/libcuda.1.dylib'

esd100 · Feb 20, 2017

@victorv Unfortunately, this problem still persists.

I am getting

(tensorflow) systems-mbp:~ test$ python
Python 3.5.2 |Anaconda 4.3.0 (x86_64)| (default, Jul  2 2016, 17:52:12) 
[GCC 4.2.1 Compatible Apple LLVM 4.2 (clang-425.0.28)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcublas.8.0.dylib locally
Segmentation fault: 11

hbfs · Feb 21, 2017

Here's a successful 10.11.6 build. Mainly for my own documentation, it has been simplified and Just Works™ :

Pre-requisites:

OS X 10.11.6
homebrew 1.1.8
Python 2.7.11
pip 9.0.1

Download and install directly from nVIDIA:

Install CUDA-8.0
Install cuDNN-5.1
a. Copy files to /Developer/NVIDIA/CUDA-8.0/{lib,include}
b. Symlink ln -s /Developer/NVIDIA/CUDA-8.0/lib/cudnn* /usr/local/cuda/lib/
c. Create symlink for libcuda.1.dylib
ln -s /usr/local/cuda/lib/libcuda.dylib /usr/local/cuda/lib/libcuda.1.dylib

Test CUDA install

Copy /Developer/NVIDIA/CUDA-8.0/samples to your working directory
cd $CUDA_SAMPLE_DIR/
a. To make all the samples: make -j<# of threads>
b. To make only the deviceQueryDrv Utility, cd to 1_Utilities/deviceQueryDrv and type make -j<# of threads>
$CUDA_SAMPLE_DIR/bin/x86_64/darwin/release/deviceQueryDrv

expected output (for my GTX 960 4GB, yours will vary depending on your GPU):

» bin/x86_64/darwin/release/deviceQueryDrv
bin/x86_64/darwin/release/deviceQueryDrv Starting...

CUDA Device Query (Driver API) statically linked version
Detected 1 CUDA Capable device(s)

Device 0: "GeForce GTX 960"
  CUDA Driver Version:                           8.0
  CUDA Capability Major/Minor version number:    5.2
  Total amount of global memory:                 4096 MBytes (4294770688 bytes)
  ( 8) Multiprocessors, (128) CUDA Cores/MP:     1024 CUDA Cores
  GPU Max Clock rate:                            1291 MHz (1.29 GHz)
  Memory Clock rate:                             3505 Mhz
  Memory Bus Width:                              128-bit
  L2 Cache Size:                                 1048576 bytes
  Max Texture Dimension Sizes                    1D=(65536) 2D=(65536, 65536) 3D=(4096, 4096, 4096)
  Maximum Layered 1D Texture Size, (num) layers  1D=(16384), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(16384, 16384), 2048 layers
  Total amount of constant memory:               65536 bytes
  Total amount of shared memory per block:       49152 bytes
  Total number of registers available per block: 65536
  Warp size:                                     32
  Maximum number of threads per multiprocessor:  2048
  Maximum number of threads per block:           1024
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
  Max dimension size of a grid size (x,y,z):    (2147483647, 65535, 65535)
  Texture alignment:                             512 bytes
  Maximum memory pitch:                          2147483647 bytes
  Concurrent copy and kernel execution:          Yes with 2 copy engine(s)
  Run time limit on kernels:                     Yes
  Integrated GPU sharing Host Memory:            No
  Support host page-locked memory mapping:       Yes
  Concurrent kernel execution:                   Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support:                        Disabled
  Device supports Unified Addressing (UVA):      Yes
  Device PCI Domain ID / Bus ID / location ID:   0 / 4 / 0
  Compute Mode:
     < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >
Result = PASS

Other software installs

You might have to install bazel.

I recommend brew install bazel.

brew while not perfect, simplifies your life. It is a simple to use package manager for OS X with a variety of packages available and maintained.

Install Tensorflow

Automagically install GPU version pip install tensorflow-gpu

Test

tensorflow-test.py:

import tensorflow as tf

# Creates a graph.
a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3], name='a')
b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2], name='b')
c = tf.matmul(a, b)

# Creates a session with log_device_placement set to True.
sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))

# Runs the op.
print sess.run(c)

expected output (the final result should be the same, the only differences being the GPU used and amount of free GPU RAM):

» python ./tensorflow-test.py
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcublas.dylib locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcudnn.dylib locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcufft.dylib locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcuda.1.dylib locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcurand.dylib locally
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:901] OS X does not support NUMA - returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 0 with properties:
name: GeForce GTX 960
major: 5 minor: 2 memoryClockRate (GHz) 1.291
pciBusID 0000:04:00.0
Total memory: 4.00GiB
Free memory: 921.46MiB
I tensorflow/core/common_runtime/gpu/gpu_device.cc:906] DMA: 0
I tensorflow/core/common_runtime/gpu/gpu_device.cc:916] 0:   Y
I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 960, pci bus id: 0000:04:00.0)
Device mapping:
/job:localhost/replica:0/task:0/gpu:0 -> device: 0, name: GeForce GTX 960, pci bus id: 0000:04:00.0
I tensorflow/core/common_runtime/direct_session.cc:255] Device mapping:
/job:localhost/replica:0/task:0/gpu:0 -> device: 0, name: GeForce GTX 960, pci bus id: 0000:04:00.0

MatMul: (MatMul): /job:localhost/replica:0/task:0/gpu:0
I tensorflow/core/common_runtime/simple_placer.cc:827] MatMul: (MatMul)/job:localhost/replica:0/task:0/gpu:0
b: (Const): /job:localhost/replica:0/task:0/gpu:0
I tensorflow/core/common_runtime/simple_placer.cc:827] b: (Const)/job:localhost/replica:0/task:0/gpu:0
a: (Const): /job:localhost/replica:0/task:0/gpu:0
I tensorflow/core/common_runtime/simple_placer.cc:827] a: (Const)/job:localhost/replica:0/task:0/gpu:0
[[ 22.  28.]
 [ 49.  64.]]

Enjoy your tensorflow install and make something insanely great.

Feedback

If this worked or more importantly, didnt work, let me know and I'll try to keep it updated.

esd100 · Feb 21, 2017

@hbfs please clarify specifics of:

Install cuDNN-5.1
a. Copy files to /Developer/NVIDIA/CUDA-8.0/{lib,include}
b. Symlink ln -s /Developer/NVIDIA/CUDA-8.0lib/cudnn* /usr/local/cuda/lib/

What do the curly brackets mean?
Why do you have a directory CUDA-8.0lib

peterswang · Feb 22, 2017

Had managed to build TF 0.12 by following How to fix dyld: Library not loaded: @rpath/libcudart.7.5.dylib issue when you build tensorflow on your Mac.

Just tried the building TF 1.0 from source using the same method and now the following error shows up after the patch in genrule-setup.sh:
ERROR: /Users/peter_wang/TF/tensorflow-1.0/tensorflow/python/BUILD:2279:1: Linking of rule '//tensorflow/python:_pywrap_tensorflow.so' failed: link_dynamic_library.sh failed: error executing command external/bazel_tools/tools/cpp/link_dynamic_library.sh no ignored ignored ignored external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -shared -o ... (remaining 648 argument(s) skipped): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1.
clang: warning: argument unused during compilation: '-pthread'
ld: file not found: -lcudart.8.0
clang: error: linker command failed with exit code 1 (use -v to see invocation)
Target //tensorflow/tools/pip_package:build_pip_package failed to build

Environment: 2012 MBP with Nvidia 650M, OS X 10.12.3, Python 2.7.13.

Links in /usr/local/cuda/lib, as suggested by @victorv was set before, though this doesn't seem to be the same issue.
lrwxr-xr-x 1 root wheel 33 Feb 10 14:30 /usr/local/cuda/lib/libcuda.1.dylib -> /usr/local/cuda/lib/libcuda.dylib
-rwxr-xr-x 1 root wheel 13504 Jan 24 11:58 /usr/local/cuda/lib/libcuda.dylib
lrwxr-xr-x@ 1 root wheel 45 Nov 3 11:40 /usr/local/cuda/lib/libcudadevrt.a -> /Developer/NVIDIA/CUDA-8.0/lib/libcudadevrt.a
lrwxr-xr-x@ 1 root wheel 50 Nov 3 11:40 /usr/local/cuda/lib/libcudart.8.0.dylib -> /Developer/NVIDIA/CUDA-8.0/lib/libcudart.8.0.dylib
lrwxr-xr-x@ 1 root wheel 46 Nov 3 11:40 /usr/local/cuda/lib/libcudart.dylib -> /Developer/NVIDIA/CUDA-8.0/lib/libcudart.dylib
lrwxr-xr-x@ 1 root wheel 49 Nov 3 11:40 /usr/local/cuda/lib/libcudart_static.a -> /Developer/NVIDIA/CUDA-8.0/lib/libcudart_static.a

"pip install tensorflow-gpu" does work. However, the standard build apparently doesn't include AVX & SSE, which I'd like to make use of as well.

victorv · Feb 22, 2017

@esd100 Do you have more detail on your segfault?

esd100 · Feb 22, 2017

@victorv Tell me how to get more detail and I'll happily provide it.

I am using:
conda 4.3.11
Python 3.5.2 :: Anaconda 4.3.0 (x86_64)

=========================================
NVIDIA CUDA PATH
export PATH=/Developer/NVIDIA/CUDA-8.0/bin${PATH:+:${PATH}}
export DYLD_LIBRARY_PATH=/Developer/NVIDIA/CUDA-8.0/lib${DYLD_LIBRARY_PATH:+:${DYLD_LIBRARY_PATH}}

Files are installed in:
/Developer/NVIDIA/CUDA-8.0/
/Developer/NVIDIA/cuDNN

=========================================

CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 1 CUDA Capable device(s)

Device 0: "GeForce GT 750M"
CUDA Driver Version / Runtime Version 8.0 / 8.0
CUDA Capability Major/Minor version number: 3.0
Total amount of global memory: 2048 MBytes (2147024896 bytes)
( 2) Multiprocessors, (192) CUDA Cores/MP: 384 CUDA Cores
GPU Max Clock rate: 926 MHz (0.93 GHz)
Memory Clock rate: 2508 Mhz
Memory Bus Width: 128-bit
L2 Cache Size: 262144 bytes
Maximum Texture Dimension Size (x,y,z) 1D=(65536), 2D=(65536, 65536), 3D=(4096, 4096, 4096)
Maximum Layered 1D Texture Size, (num) layers 1D=(16384), 2048 layers
Maximum Layered 2D Texture Size, (num) layers 2D=(16384, 16384), 2048 layers
Total amount of constant memory: 65536 bytes
Total amount of shared memory per block: 49152 bytes
Total number of registers available per block: 65536
Warp size: 32
Maximum number of threads per multiprocessor: 2048
Maximum number of threads per block: 1024
Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
Max dimension size of a grid size (x,y,z): (2147483647, 65535, 65535)
Maximum memory pitch: 2147483647 bytes
Texture alignment: 512 bytes
Concurrent copy and kernel execution: Yes with 1 copy engine(s)
Run time limit on kernels: Yes
Integrated GPU sharing Host Memory: No
Support host page-locked memory mapping: Yes
Alignment requirement for Surfaces: Yes
Device has ECC support: Disabled
Device supports Unified Addressing (UVA): Yes
Device PCI Domain ID / Bus ID / location ID: 0 / 1 / 0
Compute Mode:
< Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 8.0, CUDA Runtime Version = 8.0, NumDevs = 1, Device0 = GeForce GT 750M
Result = PASS

=========================================

$ system_profiler SPSoftwareDataType)

System Version: macOS 10.12.2 (16C67)
Kernel Version: Darwin 16.3.0
Boot Volume: Macintosh HD
Boot Mode: Normal
Secure Virtual Memory: Enabled
System Integrity Protection: Enabled

=========================================

$ pkgutil --pkg-info=com.apple.pkg.CLTools_Executables

package-id: com.apple.pkg.CLTools_Executables
version: 8.2.0.0.1.1480973914

=========================================
$ gcc --version
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/usr/include/c++/4.2.1

Apple LLVM version 8.0.0 (clang-800.0.42.1)
Target: x86_64-apple-darwin16.3.0
Thread model: posix

InstalledDir:
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

hbfs · Feb 22, 2017

@esd100 {} is for shell expansion (see 3.4.2 Brace expansion http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_03_04.html)

8.0lib was a typo, missed a '/', thanks for pointing that out. Fixed.

@peterswang good point. The pre-compiled version doesn't have all the optimization flags. For best performance for a specific system, build from source. (ubuntu vs gentoo philosophy)

The tensorflow guide on building from source is well documented: https://www.tensorflow.org/install/install_sources

victorv · Feb 22, 2017

@esd100 Segmentation fault: 11 is usually followed by a OS X crash report.

You can find crash reports in your Console application -> User Diagnostic Reports

The segfault is usually a result of mixing your link libraries.

You can avoid lots of these issues by using the docker images.

esd100 · Mar 2, 2017

@victorv Thanks. Sorry for the delay.

This is the crash report after putting in the commands:

source activate tensorflow
python
import tensorflow as tf

Process:               python3.5 [25276]
Path:                  /anaconda/*/python3.5
Identifier:            python3.5
Version:               ???
Code Type:             X86-64 (Native)
Parent Process:        bash [25151]
Responsible:           python3.5 [25276]
User ID:               501

Date/Time:             2017-03-02 18:23:28.885 -0500
OS Version:            Mac OS X 10.12.3 (16D32)
Report Version:        12
Anonymous UUID:        0F0D56D5-52EB-7CFC-CD5E-ED3D20E9E476

Sleep/Wake UUID:       6CAAD8B9-E727-4612-A71C-06E5466B2E86

Time Awake Since Boot: 19000 seconds
Time Since Wake:       4200 seconds

System Integrity Protection: enabled

Crashed Thread:        0  Dispatch queue: com.apple.main-thread

Exception Type:        EXC_BAD_ACCESS (SIGSEGV)
Exception Codes:       KERN_INVALID_ADDRESS at 0x0000000000000000
Exception Note:        EXC_CORPSE_NOTIFY

Termination Signal:    Segmentation fault: 11
Termination Reason:    Namespace SIGNAL, Code 0xb
Terminating Process:   exc handler [0]

VM Regions Near 0:
--> 
    __TEXT                 0000000100000000-0000000100001000 [    4K] r-x/rwx SM=COW  /anaconda/*/*.5

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   libsystem_c.dylib             	0x00007fffb545ab52 strlen + 18
1   _pywrap_tensorflow.so         	0x000000010537ad8a perftools::gputools::internal::DsoLoader::GetDsoHandle(tensorflow::StringPiece, void**, perftools::gputools::internal::DsoLoader::LoadKind) + 282
2   _pywrap_tensorflow.so         	0x000000010537b69c perftools::gputools::internal::DsoLoader::GetCudnnDsoHandle(void**) + 204
3   _pywrap_tensorflow.so         	0x000000010537ccb2 std::__1::__function::__func<tensorflow::Status (*)(void**), std::__1::allocator<tensorflow::Status (*)(void**)>, tensorflow::Status (void**)>::operator()(void**&&) + 18
4   _pywrap_tensorflow.so         	0x000000010537c45f perftools::gputools::internal::CachedDsoLoader::FetchHandleResult(std::__1::function<tensorflow::Status (void**)>) + 47
5   _pywrap_tensorflow.so         	0x000000010537c6ff perftools::gputools::internal::CachedDsoLoader::GetCudnnDsoHandle() + 111
6   _pywrap_tensorflow.so         	0x0000000105341e62 perftools::gputools::initialize_cudnn() + 242
7   dyld                          	0x00007fff6974e75f ImageLoaderMachO::doModInitFunctions(ImageLoader::LinkContext const&) + 385
8   dyld                          	0x00007fff6974e962 ImageLoaderMachO::doInitialization(ImageLoader::LinkContext const&) + 40
9   dyld                          	0x00007fff6974a1ee ImageLoader::recursiveInitialization(ImageLoader::LinkContext const&, unsigned int, char const*, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 338
10  dyld                          	0x00007fff69749268 ImageLoader::processInitializers(ImageLoader::LinkContext const&, unsigned int, ImageLoader::InitializerTimingList&, ImageLoader::UninitedUpwards&) + 138
11  dyld                          	0x00007fff697492fd ImageLoader::runInitializers(ImageLoader::LinkContext const&, ImageLoader::InitializerTimingList&) + 75
12  dyld                          	0x00007fff6973e70d dyld::runInitializers(ImageLoader*) + 87
13  dyld                          	0x00007fff69746150 dlopen + 556
14  libdyld.dylib                 	0x00007fffb5421a3e dlopen + 59
15  libpython3.5m.dylib           	0x00000001000f9f17 _PyImport_FindSharedFuncptr + 311
16  libpython3.5m.dylib           	0x00000001000db506 _PyImport_LoadDynamicModuleWithSpec + 422
17  libpython3.5m.dylib           	0x00000001000db0c2 _imp_create_dynamic + 258
18  libpython3.5m.dylib           	0x000000010004ff38 PyCFunction_Call + 280
19  libpython3.5m.dylib           	0x00000001000bde4b PyEval_EvalFrameEx + 25355
20  libpython3.5m.dylib           	0x00000001000c10c3 _PyEval_EvalCodeWithName + 1779
21  libpython3.5m.dylib           	0x00000001000c19ae fast_function + 334
22  libpython3.5m.dylib           	0x00000001000bd434 PyEval_EvalFrameEx + 22772
23  libpython3.5m.dylib           	0x00000001000c192f fast_function + 207
24  libpython3.5m.dylib           	0x00000001000bd434 PyEval_EvalFrameEx + 22772
25  libpython3.5m.dylib           	0x00000001000c192f fast_function + 207
26  libpython3.5m.dylib           	0x00000001000bd434 PyEval_EvalFrameEx + 22772
27  libpython3.5m.dylib           	0x00000001000c192f fast_function + 207
28  libpython3.5m.dylib           	0x00000001000bd434 PyEval_EvalFrameEx + 22772
29  libpython3.5m.dylib           	0x00000001000c192f fast_function + 207
30  libpython3.5m.dylib           	0x00000001000bd434 PyEval_EvalFrameEx + 22772
31  libpython3.5m.dylib           	0x00000001000c10c3 _PyEval_EvalCodeWithName + 1779
32  libpython3.5m.dylib           	0x00000001000c19ae fast_function + 334
33  libpython3.5m.dylib           	0x00000001000bd434 PyEval_EvalFrameEx + 22772
34  libpython3.5m.dylib           	0x00000001000c192f fast_function + 207
35  libpython3.5m.dylib           	0x00000001000bd434 PyEval_EvalFrameEx + 22772
36  libpython3.5m.dylib           	0x00000001000c192f fast_function + 207
37  libpython3.5m.dylib           	0x00000001000bd434 PyEval_EvalFrameEx + 22772
38  libpython3.5m.dylib           	0x00000001000c10c3 _PyEval_EvalCodeWithName + 1779
39  libpython3.5m.dylib           	0x00000001000b7ac1 PyEval_EvalCode + 81
40  libpython3.5m.dylib           	0x00000001000b52ab builtin_exec + 555
41  libpython3.5m.dylib           	0x000000010004ff38 PyCFunction_Call + 280
42  libpython3.5m.dylib           	0x00000001000bde4b PyEval_EvalFrameEx + 25355
43  libpython3.5m.dylib           	0x00000001000c10c3 _PyEval_EvalCodeWithName + 1779
44  libpython3.5m.dylib           	0x00000001000c19ae fast_function + 334
45  libpython3.5m.dylib           	0x00000001000bd434 PyEval_EvalFrameEx + 22772
46  libpython3.5m.dylib           	0x00000001000c192f fast_function + 207
47  libpython3.5m.dylib           	0x00000001000bd434 PyEval_EvalFrameEx + 22772
48  libpython3.5m.dylib           	0x00000001000c192f fast_function + 207
49  libpython3.5m.dylib           	0x00000001000bd434 PyEval_EvalFrameEx + 22772
50  libpython3.5m.dylib           	0x00000001000c192f fast_function + 207
51  libpython3.5m.dylib           	0x00000001000bd434 PyEval_EvalFrameEx + 22772
52  libpython3.5m.dylib           	0x00000001000c10c3 _PyEval_EvalCodeWithName + 1779
53  libpython3.5m.dylib           	0x00000001000b7b1e PyEval_EvalCodeEx + 78
54  libpython3.5m.dylib           	0x000000010003430f function_call + 351
55  libpython3.5m.dylib           	0x000000010000fd73 PyObject_Call + 99
56  libpython3.5m.dylib           	0x0000000100010815 _PyObject_CallMethodIdObjArgs + 421
57  libpython3.5m.dylib           	0x00000001000da3af PyImport_ImportModuleLevelObject + 1951
58  libpython3.5m.dylib           	0x00000001000b4854 builtin___import__ + 132
59  libpython3.5m.dylib           	0x000000010004fe5c PyCFunction_Call + 60
60  libpython3.5m.dylib           	0x00000001000bde4b PyEval_EvalFrameEx + 25355
61  libpython3.5m.dylib           	0x00000001000c10c3 _PyEval_EvalCodeWithName + 1779
62  libpython3.5m.dylib           	0x00000001000c19ae fast_function + 334
63  libpython3.5m.dylib           	0x00000001000bd434 PyEval_EvalFrameEx + 22772
64  libpython3.5m.dylib           	0x00000001000c10c3 _PyEval_EvalCodeWithName + 1779
65  libpython3.5m.dylib           	0x00000001000b7b1e PyEval_EvalCodeEx + 78
66  libpython3.5m.dylib           	0x000000010003430f function_call + 351
67  libpython3.5m.dylib           	0x000000010000fd73 PyObject_Call + 99
68  libpython3.5m.dylib           	0x0000000100010815 _PyObject_CallMethodIdObjArgs + 421
69  libpython3.5m.dylib           	0x00000001000da4f4 PyImport_ImportModuleLevelObject + 2276
70  libpython3.5m.dylib           	0x00000001000b4854 builtin___import__ + 132
71  libpython3.5m.dylib           	0x000000010004fe5c PyCFunction_Call + 60
72  libpython3.5m.dylib           	0x000000010000fd73 PyObject_Call + 99
73  libpython3.5m.dylib           	0x00000001000bbd2a PyEval_EvalFrameEx + 16874
74  libpython3.5m.dylib           	0x00000001000c10c3 _PyEval_EvalCodeWithName + 1779
75  libpython3.5m.dylib           	0x00000001000b7ac1 PyEval_EvalCode + 81
76  libpython3.5m.dylib           	0x00000001000b52ab builtin_exec + 555
77  libpython3.5m.dylib           	0x000000010004ff38 PyCFunction_Call + 280
78  libpython3.5m.dylib           	0x00000001000bde4b PyEval_EvalFrameEx + 25355
79  libpython3.5m.dylib           	0x00000001000c10c3 _PyEval_EvalCodeWithName + 1779
80  libpython3.5m.dylib           	0x00000001000c19ae fast_function + 334
81  libpython3.5m.dylib           	0x00000001000bd434 PyEval_EvalFrameEx + 22772
82  libpython3.5m.dylib           	0x00000001000c192f fast_function + 207
83  libpython3.5m.dylib           	0x00000001000bd434 PyEval_EvalFrameEx + 22772
84  libpython3.5m.dylib           	0x00000001000c192f fast_function + 207
85  libpython3.5m.dylib           	0x00000001000bd434 PyEval_EvalFrameEx + 22772
86  libpython3.5m.dylib           	0x00000001000c192f fast_function + 207
87  libpython3.5m.dylib           	0x00000001000bd434 PyEval_EvalFrameEx + 22772
88  libpython3.5m.dylib           	0x00000001000c10c3 _PyEval_EvalCodeWithName + 1779
89  libpython3.5m.dylib           	0x00000001000b7b1e PyEval_EvalCodeEx + 78
90  libpython3.5m.dylib           	0x000000010003430f function_call + 351
91  libpython3.5m.dylib           	0x000000010000fd73 PyObject_Call + 99
92  libpython3.5m.dylib           	0x0000000100010815 _PyObject_CallMethodIdObjArgs + 421
93  libpython3.5m.dylib           	0x00000001000da3af PyImport_ImportModuleLevelObject + 1951
94  libpython3.5m.dylib           	0x00000001000b4854 builtin___import__ + 132
95  libpython3.5m.dylib           	0x000000010004fe5c PyCFunction_Call + 60
96  libpython3.5m.dylib           	0x000000010000fd73 PyObject_Call + 99
97  libpython3.5m.dylib           	0x00000001000bbd2a PyEval_EvalFrameEx + 16874
98  libpython3.5m.dylib           	0x00000001000c10c3 _PyEval_EvalCodeWithName + 1779
99  libpython3.5m.dylib           	0x00000001000b7ac1 PyEval_EvalCode + 81
100 libpython3.5m.dylib           	0x00000001000b52ab builtin_exec + 555
101 libpython3.5m.dylib           	0x000000010004ff38 PyCFunction_Call + 280
102 libpython3.5m.dylib           	0x00000001000bde4b PyEval_EvalFrameEx + 25355
103 libpython3.5m.dylib           	0x00000001000c10c3 _PyEval_EvalCodeWithName + 1779
104 libpython3.5m.dylib           	0x00000001000c19ae fast_function + 334
105 libpython3.5m.dylib           	0x00000001000bd434 PyEval_EvalFrameEx + 22772
106 libpython3.5m.dylib           	0x00000001000c192f fast_function + 207
107 libpython3.5m.dylib           	0x00000001000bd434 PyEval_EvalFrameEx + 22772
108 libpython3.5m.dylib           	0x00000001000c192f fast_function + 207
109 libpython3.5m.dylib           	0x00000001000bd434 PyEval_EvalFrameEx + 22772
110 libpython3.5m.dylib           	0x00000001000c192f fast_function + 207
111 libpython3.5m.dylib           	0x00000001000bd434 PyEval_EvalFrameEx + 22772
112 libpython3.5m.dylib           	0x00000001000c10c3 _PyEval_EvalCodeWithName + 1779
113 libpython3.5m.dylib           	0x00000001000b7b1e PyEval_EvalCodeEx + 78
114 libpython3.5m.dylib           	0x000000010003430f function_call + 351
115 libpython3.5m.dylib           	0x000000010000fd73 PyObject_Call + 99
116 libpython3.5m.dylib           	0x0000000100010815 _PyObject_CallMethodIdObjArgs + 421
117 libpython3.5m.dylib           	0x00000001000da3af PyImport_ImportModuleLevelObject + 1951
118 libpython3.5m.dylib           	0x00000001000b4854 builtin___import__ + 132
119 libpython3.5m.dylib           	0x000000010004fe5c PyCFunction_Call + 60
120 libpython3.5m.dylib           	0x000000010000fd73 PyObject_Call + 99
121 libpython3.5m.dylib           	0x00000001000bbd2a PyEval_EvalFrameEx + 16874
122 libpython3.5m.dylib           	0x00000001000c10c3 _PyEval_EvalCodeWithName + 1779
123 libpython3.5m.dylib           	0x00000001000b7ac1 PyEval_EvalCode + 81
124 libpython3.5m.dylib           	0x00000001000e642c PyRun_InteractiveOneObject + 588
125 libpython3.5m.dylib           	0x00000001000e5d5e PyRun_InteractiveLoopFlags + 206
126 libpython3.5m.dylib           	0x00000001000e5c5c PyRun_AnyFileExFlags + 60
127 libpython3.5m.dylib           	0x00000001000fcd47 Py_Main + 3591
128 python                        	0x0000000100000dc7 main + 215
129 python                        	0x0000000100000ce4 start + 52

Thread 0 crashed with X86 Thread State (64-bit):
  rax: 0x0000000000000000  rbx: 0x0000000000000000  rcx: 0x0000000000000000  rdx: 0x0000000000000000
  rdi: 0x0000000000000000  rsi: 0x0000000000000307  rbp: 0x00007fff5bff8e20  rsp: 0x00007fff5bff8e20
   r8: 0x00000000fffffff1   r9: 0x0000000101202778  r10: 0x000000000000000f  r11: 0x0000000000000000
  r12: 0x00007fff5bff9209  r13: 0xf7bee2bde4bf00e0  r14: 0x00007fff5bff9270  r15: 0x0000000000000010
  rip: 0x00007fffb545ab52  rfl: 0x0000000000010246  cr2: 0x0000000000000000
  
Logical CPU:     6
Error Code:      0x00000004
Trap Number:     14


Binary Images:
       0x100000000 -        0x100000ff7 +python (???) <A92DBDA7-44FF-35CF-8860-69BA977B417E> /anaconda/*/python
       0x100003000 -        0x1001b5fff +libpython3.5m.dylib (3.5) <D24D7ED3-029D-3C55-8431-04B5A2E8639A> /anaconda/*/libpython3.5m.dylib
       0x1003e6000 -        0x1003e7fff +_heapq.so (???) <01E9935D-050D-3732-BB3C-1F1A941DED4D> /anaconda/*/_heapq.so
       0x1003eb000 -        0x1003edfff +readline.so (???) <2BBC4898-3D25-3E58-BED2-4CA01365FFB9> /anaconda/*/readline.so
       0x1003f5000 -        0x1003f5ff7 +_opcode.so (???) <8F431933-E6A7-3318-81E4-C8157F1DA7F6> /anaconda/*/_opcode.so
       0x1003f9000 -        0x1003f9fff +grp.so (???) <75A30066-2CB2-3F6E-ADAC-B249233CDB42> /anaconda/*/grp.so
       0x1003fc000 -        0x1003fdfff +_random.so (???) <3326504F-74E4-33F3-8D0E-0468B6480468> /anaconda/*/_random.so
       0x100731000 -        0x100762fff +libreadline.6.2.dylib (6.2) <8B0801EE-2449-E9D6-850B-4E7FB34BF7FB> /anaconda/*/libreadline.6.2.dylib
       0x100778000 -        0x10078aff7 +_ctypes.so (???) <6784BCB6-23E1-3728-920D-D75D0AC4F34F> /anaconda/*/_ctypes.so
       0x10079b000 -        0x10079ffff +_struct.so (???) <42353FCD-910E-319F-8BD5-F4A5BD9AAF2E> /anaconda/*/_struct.so
       0x1007e8000 -        0x1007edff7 +math.so (???) <6301B65C-26E9-3CF7-9013-1239B15EF4D3> /anaconda/*/math.so
       0x1007f4000 -        0x1007f7ff7 +_lzma.so (???) <45350179-45D9-3E0C-9034-A863F5978360> /anaconda/*/_lzma.so
       0x101412000 -        0x1015a1fff +multiarray.cpython-35m-darwin.so (???) <C3082A47-3D27-3214-A3D3-346C8111EDE2> /anaconda/*/multiarray.cpython-35m-darwin.so
       0x10169f000 -        0x1016abfff +_datetime.so (???) <2264E487-677B-34D3-A546-9745721617A1> /anaconda/*/_datetime.so
       0x1016b6000 -        0x101789fff +umath.cpython-35m-darwin.so (???) <4F1ED886-054E-379D-96EE-5A522B1AD8FA> /anaconda/*/umath.cpython-35m-darwin.so
       0x1017c2000 -        0x1017d1fff +_pickle.so (???) <76465AD0-A0A7-311E-BDAE-09A7768E8860> /anaconda/*/_pickle.so
       0x1017dd000 -        0x1017f5ff7 +_bz2.so (???) <F29CB3C3-69BD-340E-AC38-BAA8C8F87D51> /anaconda/*/_bz2.so
       0x1017fb000 -        0x1017fcff7 +lapack_lite.cpython-35m-darwin.so (???) <EAAAB223-4E4A-3E1D-B0CE-E8A639265D3C> /anaconda/*/lapack_lite.cpython-35m-darwin.so
       0x102a80000 -        0x102ab4fff +liblzma.5.dylib (8.2) <DCF7E34A-AED3-3DB8-8DD8-F17522DAAE42> /anaconda/*/liblzma.5.dylib
       0x102afb000 -        0x102afdff7 +_hashlib.so (???) <2A1F4AD5-F0A5-3636-B93A-87A1158EEDB7> /anaconda/*/_hashlib.so
       0x102b02000 -        0x102b4cff7 +libssl.1.0.0.dylib (0) <9F42DCB5-702D-327C-84C1-AB64D6F942E1> /anaconda/*/libssl.1.0.0.dylib
       0x102b69000 -        0x102cea227 +libcrypto.1.0.0.dylib (0) <F7749773-18B9-33DE-B4BB-A9D1BFB6D96E> /anaconda/*/libcrypto.1.0.0.dylib
       0x102e20000 -        0x102e38fff +_umath_linalg.cpython-35m-darwin.so (???) <3DDEBEDD-5C80-339D-9770-6D62514FAA06> /anaconda/*/_umath_linalg.cpython-35m-darwin.so
       0x102ec6000 -        0x102ecfff7 +fftpack_lite.cpython-35m-darwin.so (???) <58E02D71-415E-36AA-9D54-3F32E4786163> /anaconda/*/fftpack_lite.cpython-35m-darwin.so
       0x103040000 -        0x1030eefff +mtrand.cpython-35m-darwin.so (???) <BE72B37C-5265-353B-AD9B-A44E9E57640D> /anaconda/*/mtrand.cpython-35m-darwin.so
       0x10321c000 -        0x10637d767 +_pywrap_tensorflow.so (0) <CFBCCB99-C59C-389E-85E0-DC9B3AE13CA8> /anaconda/*/_pywrap_tensorflow.so
       0x112a39000 -        0x112a74fff +libcudart.8.0.dylib (0) <6E55D67C-1A95-3C2B-B2C1-C3F5827A3F03> /Developer/NVIDIA/CUDA-8.0/lib/libcudart.8.0.dylib
       0x113000000 -        0x11333aff7 +libcublas.8.0.dylib (0) <373C3010-E15D-3563-B655-76532180F4CF> /Developer/NVIDIA/CUDA-8.0/lib/libcublas.8.0.dylib
    0x7fff69739000 -     0x7fff69776267  dyld (421.2) <947FC440-80F9-32F7-A773-6FC418FE1AB7> /usr/lib/dyld
    0x7fff9c9d9000 -     0x7fff9c9d9fff  com.apple.Accelerate (1.11 - Accelerate 1.11) <D700DBDF-69AE-37A2-B9C7-0961CF0B6841> /System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate
    0x7fff9c9f2000 -     0x7fff9cf0bfeb  com.apple.vImage (8.1 - ???) <6408805B-67E9-3874-8D32-0BB814CE5CDA> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vImage.framework/Versions/A/vImage
    0x7fff9cf0c000 -     0x7fff9d07cff3  libBLAS.dylib (1185) <C7E42BBE-2337-3AEF-9C45-A2F2CB1A5B3E> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
    0x7fff9d07d000 -     0x7fff9d091ffb  libBNNS.dylib (14) <CFDEE88D-E002-347C-BC68-83099651585B> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBNNS.dylib
    0x7fff9d092000 -     0x7fff9d488fef  libLAPACK.dylib (1185) <2E8201CB-9A41-3D65-853E-841917FCE77B> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libLAPACK.dylib
    0x7fff9d489000 -     0x7fff9d49ffff  libLinearAlgebra.dylib (1185) <8CC29DE1-A231-3D5E-B5F1-DCC309036FE0> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libLinearAlgebra.dylib
    0x7fff9d4a0000 -     0x7fff9d4a6fff  libQuadrature.dylib (3) <120F6228-A3D4-3184-89D7-785ADC2AC715> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libQuadrature.dylib
    0x7fff9d4a7000 -     0x7fff9d4bbff7  libSparseBLAS.dylib (1185) <C35235B7-CFA6-39A7-BD6E-79F4D9CAFD36> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libSparseBLAS.dylib
    0x7fff9d4bc000 -     0x7fff9d643fe7  libvDSP.dylib (600) <F59348AA-E1D3-3A27-8AB5-F546D38BFB76> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libvDSP.dylib
    0x7fff9d644000 -     0x7fff9d6f6ffb  libvMisc.dylib (600) <70D4B548-47EE-3C6B-A93B-3EA6B60701E0> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libvMisc.dylib
    0x7fff9d6f7000 -     0x7fff9d6f7fff  com.apple.Accelerate.vecLib (3.11 - vecLib 3.11) <A395B521-8E54-30F2-B4FE-355D68900DAF> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/vecLib
    0x7fff9fe45000 -     0x7fffa02dffff  com.apple.CoreFoundation (6.9 - 1348.28) <A40AA224-7A50-3989-95D0-5A228A0E2FAF> /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
    0x7fffa1e27000 -     0x7fffa1ebcff7  com.apple.framework.IOKit (2.0.2 - 1324.30.13) <163BE7FA-B29A-348F-8B5F-E301F2E8C964> /System/Library/Frameworks/IOKit.framework/Versions/A/IOKit
    0x7fffa634b000 -     0x7fffa666afff  com.apple.security (7.0 - 57740.31.2) <A47D7BAE-0591-3184-8E44-FB2EB08A19C2> /System/Library/Frameworks/Security.framework/Versions/A/Security
    0x7fffb3c42000 -     0x7fffb3c43ff3  libDiagnosticMessagesClient.dylib (102) <422911A4-E273-3E88-BFC4-DF6470E48242> /usr/lib/libDiagnosticMessagesClient.dylib
    0x7fffb3e7b000 -     0x7fffb3e7bfff  libOpenScriptingUtil.dylib (172) <D025E180-BB3B-3FFA-98FC-B6835354D723> /usr/lib/libOpenScriptingUtil.dylib
    0x7fffb3e81000 -     0x7fffb3e82ff3  libSystem.B.dylib (1238) <9CB018AF-54E9-300F-82BE-81FE553C9154> /usr/lib/libSystem.B.dylib
    0x7fffb3f9a000 -     0x7fffb3f9aff3  libauto.dylib (187) <5BBF6A00-CC76-389D-84E7-CA88EDADE683> /usr/lib/libauto.dylib
    0x7fffb3f9b000 -     0x7fffb3fabff3  libbsm.0.dylib (34) <20084796-B04D-3B35-A003-EA11459557A9> /usr/lib/libbsm.0.dylib
    0x7fffb3fac000 -     0x7fffb3fbaff7  libbz2.1.0.dylib (38) <6FD3B63F-0F86-3A25-BD5B-E243F58792C9> /usr/lib/libbz2.1.0.dylib
    0x7fffb3fbb000 -     0x7fffb4011ff7  libc++.1.dylib (307.4) <BEE86868-F831-384C-919E-2B286ACFE87C> /usr/lib/libc++.1.dylib
    0x7fffb4012000 -     0x7fffb403cfff  libc++abi.dylib (307.2) <1CEF8ABB-7E6D-3C2F-8E0A-E7884478DD23> /usr/lib/libc++abi.dylib
    0x7fffb4064000 -     0x7fffb4064ff7  libcoretls.dylib (121.31.1) <BCC32537-4831-3E9F-876E-8C9F4CF52FD3> /usr/lib/libcoretls.dylib
    0x7fffb4065000 -     0x7fffb4066ff3  libcoretls_cfhelpers.dylib (121.31.1) <6F37C5AD-7999-3D31-A52F-7AEED935F32D> /usr/lib/libcoretls_cfhelpers.dylib
    0x7fffb4471000 -     0x7fffb4471fff  libenergytrace.dylib (15) <A1B040A2-7977-3097-9ADF-34FF181EB970> /usr/lib/libenergytrace.dylib
    0x7fffb457a000 -     0x7fffb479fffb  libicucore.A.dylib (57149.0.1) <6B5FDA93-AA88-318F-9608-C2A33D602EC7> /usr/lib/libicucore.A.dylib
    0x7fffb47a7000 -     0x7fffb47c0ffb  liblzma.5.dylib (10) <44BD0279-99DD-36B5-8A6E-C11432E2098D> /usr/lib/liblzma.5.dylib
    0x7fffb4a80000 -     0x7fffb4ab1ffb  libncurses.5.4.dylib (51.30.1) <A8C8F837-86A2-3EC2-B2D2-6E8A267847E4> /usr/lib/libncurses.5.4.dylib
    0x7fffb4b2c000 -     0x7fffb4efcd97  libobjc.A.dylib (706) <F9AFE665-A3A2-3285-9495-19803A565861> /usr/lib/libobjc.A.dylib
    0x7fffb4eff000 -     0x7fffb4f03fff  libpam.2.dylib (21.30.1) <71EB0D88-DE84-3C8D-A2C5-58AA282BC5BC> /usr/lib/libpam.2.dylib
    0x7fffb4fbe000 -     0x7fffb5106fe3  libsqlite3.dylib (253) <B5BA5C96-AB13-34A0-8237-DD52A0181DFE> /usr/lib/libsqlite3.dylib
    0x7fffb51fb000 -     0x7fffb5208fff  libxar.1.dylib (357) <58BFB84B-66FE-3299-AA3D-BBA178ADEE39> /usr/lib/libxar.1.dylib
    0x7fffb520c000 -     0x7fffb52fbffb  libxml2.2.dylib (30.11) <E12AF929-0FA5-3214-840F-C81E6AC9F36E> /usr/lib/libxml2.2.dylib
    0x7fffb5326000 -     0x7fffb5337ff3  libz.1.dylib (67) <46E3FFA2-4328-327A-8D34-A03E20BFFB8E> /usr/lib/libz.1.dylib
    0x7fffb5346000 -     0x7fffb534aff7  libcache.dylib (79) <0C8092D3-600F-3ADD-A036-F225B6CDCA43> /usr/lib/system/libcache.dylib
    0x7fffb534b000 -     0x7fffb5356ff7  libcommonCrypto.dylib (60092.30.2) <B16E29B6-EC8D-3A8F-9A89-DD9CF35F7C4B> /usr/lib/system/libcommonCrypto.dylib
    0x7fffb5357000 -     0x7fffb535efff  libcompiler_rt.dylib (62) <E992E8D9-037C-3454-A366-A25E4D31D6BB> /usr/lib/system/libcompiler_rt.dylib
    0x7fffb535f000 -     0x7fffb5367fff  libcopyfile.dylib (138) <64E285D9-5485-333B-AEE7-8B0C8FB9275F> /usr/lib/system/libcopyfile.dylib
    0x7fffb5368000 -     0x7fffb53ebfdf  libcorecrypto.dylib (442.30.20) <2074B932-FD79-30A9-8E90-AF25C49F2AF1> /usr/lib/system/libcorecrypto.dylib
    0x7fffb53ec000 -     0x7fffb541efff  libdispatch.dylib (703.30.5) <EA0CC14E-D559-3802-B4B2-0E8C7579AAC4> /usr/lib/system/libdispatch.dylib
    0x7fffb541f000 -     0x7fffb5424ff3  libdyld.dylib (421.2) <6F506653-FFF6-3DB8-84F1-109AE3C52F32> /usr/lib/system/libdyld.dylib
    0x7fffb5425000 -     0x7fffb5425ffb  libkeymgr.dylib (28) <1A318923-1200-3B06-B432-5007D82F195D> /usr/lib/system/libkeymgr.dylib
    0x7fffb5426000 -     0x7fffb5432ffb  libkxld.dylib (3789.41.3) <87550136-9353-348B-9CD9-C342B48C5AAF> /usr/lib/system/libkxld.dylib
    0x7fffb5433000 -     0x7fffb5433fff  liblaunch.dylib (972.30.7) <15FACC21-079A-3BDF-9AFB-4253EFDEB587> /usr/lib/system/liblaunch.dylib
    0x7fffb5434000 -     0x7fffb5439fff  libmacho.dylib (894) <A2F38EC1-C37C-3B93-B0E4-36B07C177F8C> /usr/lib/system/libmacho.dylib
    0x7fffb543a000 -     0x7fffb543cff3  libquarantine.dylib (85) <C1D7749F-5F5F-3BB9-BEFC-1F0B9DA941FD> /usr/lib/system/libquarantine.dylib
    0x7fffb543d000 -     0x7fffb543effb  libremovefile.dylib (45) <CD42974E-BE0B-39FC-9BFC-8A7540A04DC6> /usr/lib/system/libremovefile.dylib
    0x7fffb543f000 -     0x7fffb5457ff7  libsystem_asl.dylib (349.30.2) <EFAC72D7-CB13-3DF7-ADF3-EC6635C6F1EA> /usr/lib/system/libsystem_asl.dylib
    0x7fffb5458000 -     0x7fffb5458ff7  libsystem_blocks.dylib (67) <B8C3701D-5A91-3D35-999D-2DC8D5393525> /usr/lib/system/libsystem_blocks.dylib
    0x7fffb5459000 -     0x7fffb54e6fef  libsystem_c.dylib (1158.30.7) <2F881962-03CB-3B9D-A782-D98C1BBA4E3D> /usr/lib/system/libsystem_c.dylib
    0x7fffb54e7000 -     0x7fffb54eaffb  libsystem_configuration.dylib (888.30.2) <4FE3983C-E4ED-3939-A578-03AD29C99788> /usr/lib/system/libsystem_configuration.dylib
    0x7fffb54eb000 -     0x7fffb54eefff  libsystem_coreservices.dylib (41.4) <1A572B9E-0C47-320F-8C64-7990D0A5FB5A> /usr/lib/system/libsystem_coreservices.dylib
    0x7fffb54ef000 -     0x7fffb5507ff3  libsystem_coretls.dylib (121.31.1) <4676F06D-274D-31BE-B61C-4D7A4AEF4858> /usr/lib/system/libsystem_coretls.dylib
    0x7fffb5508000 -     0x7fffb550efff  libsystem_dnssd.dylib (765.30.11) <DC708D84-ED7D-3936-B996-A67C66B8DDAA> /usr/lib/system/libsystem_dnssd.dylib
    0x7fffb550f000 -     0x7fffb5538ff7  libsystem_info.dylib (503.30.1) <9ED9121C-F111-3FAD-BC2F-C95DEE1C9362> /usr/lib/system/libsystem_info.dylib
    0x7fffb5539000 -     0x7fffb555bff7  libsystem_kernel.dylib (3789.41.3) <B75B128C-7D7A-3318-91CD-82B5A69C5329> /usr/lib/system/libsystem_kernel.dylib
    0x7fffb555c000 -     0x7fffb55a3fe7  libsystem_m.dylib (3121.4) <266DB92B-A86F-3691-80FB-1B26AD73CFF3> /usr/lib/system/libsystem_m.dylib
    0x7fffb55a4000 -     0x7fffb55c2ff7  libsystem_malloc.dylib (116.30.3) <F40DEE3B-386A-3529-A3F7-98117ED55BF4> /usr/lib/system/libsystem_malloc.dylib
    0x7fffb55c3000 -     0x7fffb561affb  libsystem_network.dylib (856.30.16) <4AE368E9-605D-379D-B04C-2AC7455B8250> /usr/lib/system/libsystem_network.dylib
    0x7fffb561b000 -     0x7fffb5624ff3  libsystem_networkextension.dylib (563.30.15) <EB020B0C-7DF0-3EEF-8E3C-15DA3C01D687> /usr/lib/system/libsystem_networkextension.dylib
    0x7fffb5625000 -     0x7fffb562eff3  libsystem_notify.dylib (165.20.1) <E7FD3A7C-DD07-36E2-9FA4-7561F9F114DA> /usr/lib/system/libsystem_notify.dylib
    0x7fffb562f000 -     0x7fffb5637fe7  libsystem_platform.dylib (126.1.2) <3CA06D4E-C00A-36DE-AA65-3A390097D1F6> /usr/lib/system/libsystem_platform.dylib
    0x7fffb5638000 -     0x7fffb5642ff7  libsystem_pthread.dylib (218.30.1) <C869ED7C-BE29-3532-8E69-3A8DA1447EDC> /usr/lib/system/libsystem_pthread.dylib
    0x7fffb5643000 -     0x7fffb5646ff7  libsystem_sandbox.dylib (592.31.1) <7BBFDF96-293F-3DD9-B3A4-7C168280B441> /usr/lib/system/libsystem_sandbox.dylib
    0x7fffb5647000 -     0x7fffb5648fff  libsystem_secinit.dylib (24) <5C1F1E47-0F7D-3E25-8DEB-D9DB1F902281> /usr/lib/system/libsystem_secinit.dylib
    0x7fffb5649000 -     0x7fffb5650fff  libsystem_symptoms.dylib (532.30.6) <5D990CF5-B58F-39F7-B375-99B4EC62CFBD> /usr/lib/system/libsystem_symptoms.dylib
    0x7fffb5651000 -     0x7fffb5671ff7  libsystem_trace.dylib (518.30.7) <6D34D1EA-2A3C-3D2D-803E-A666E6AEEE52> /usr/lib/system/libsystem_trace.dylib
    0x7fffb5672000 -     0x7fffb5677ffb  libunwind.dylib (35.3) <9F7C2AD8-A9A7-3DE4-828D-B0F0F166AAA0> /usr/lib/system/libunwind.dylib
    0x7fffb5678000 -     0x7fffb56a1ff7  libxpc.dylib (972.30.7) <65E41BB6-EBD5-3D93-B0BE-B190CEE4DD93> /usr/lib/system/libxpc.dylib

External Modification Summary:
  Calls made by other processes targeting this process:
    task_for_pid: 0
    thread_create: 0
    thread_set_state: 0
  Calls made by this process:
    task_for_pid: 0
    thread_create: 0
    thread_set_state: 0
  Calls made by all processes on this machine:
    task_for_pid: 12969
    thread_create: 0
    thread_set_state: 0

VM Region Summary:
ReadOnly portion of Libraries: Total=272.4M resident=0K(0%) swapped_out_or_unallocated=272.4M(100%)
Writable regions: Total=60.4M written=0K(0%) resident=0K(0%) swapped_out=0K(0%) unallocated=60.4M(100%)
 
                                VIRTUAL   REGION 
REGION TYPE                        SIZE    COUNT (non-coalesced) 
===========                     =======  ======= 
Kernel Alloc Once                    8K        2 
MALLOC                            43.4M       24 
MALLOC guard page                   16K        4 
MALLOC_LARGE (reserved)            512K        3         reserved VM address space (unallocated)
VM_ALLOCATE                          8K        3 
__DATA                            6396K      106 
__LINKEDIT                       177.8M       31 
__NV_CUDA                        169.4M        3 
__TEXT                            94.6M       99 
__UNICODE                          556K        2 
__UNIXSTACK                       16.0M        2 
shared memory                       12K        4 
===========                     =======  ======= 
TOTAL                            508.6M      271 
TOTAL, minus reserved VM space   508.1M      271

yaroslavvb · Mar 2, 2017

Have you tried setting LD_LIBRARY_PATH?
There's also this issue which tracked down similar "strlen + 18" to getenv LD_LIBRARY_PATH returning null

I actually see similar strlen + 18 with probability 50% on latest MacOS when I create session in GPU TensorFlow with CUDA_VISIBLE_DEVICES set to empty.

Most likely thing is that there's a piece of code which isn't checking for null pointers, which can happen non-deterministically. For instance on MacOS, CFStringGetCStringPtr can return null non-deterministically. That output isn't checked for null and I fixed one such place in #3448 but I see other return values that aren't checked for null, such as output of CFDictionaryGetValue in this line

victorv · Mar 3, 2017

@esd100 it could be LD_LIBRARY_PATH as @yaroslavvb has noted or you could be missing a cuda.1.dylib link in your /usr/local/cuda/lib directory.

lrwxr-xr-x 1 root staff 33 Feb 16 12:33 libcuda.1.dylib -> /usr/local/cuda/lib/libcuda.dylib

ln -s libcuda.dylib /usr/local/cuda/lib/libcuda.1.dylib

esd100 · Mar 7, 2017

@victorv @yaroslavvb I have the cuda.1.dylib link.

Would you recommend setting LD_LIBRARY_PATH in the bash profile? What do you set it to?

In my bash_profile, I have:

# NVIDIA CUDA PATH
export PATH=/Developer/NVIDIA/CUDA-8.0/bin${PATH:+:${PATH}}
export DYLD_LIBRARY_PATH=/Developer/NVIDIA/CUDA-8.0/lib${DYLD_LIBRARY_PATH:+:${DYLD_LIBRARY_PATH}}

yaroslavvb · Mar 7, 2017

export DYLD_LIBRARY_PATH=/usr/local/cuda/lib:/usr/local/cuda/extras/CUPTI/lib
export LD_LIBRARY_PATH=$DYLD_LIBRARY_PATH
export PATH=$DYLD_LIBRARY_PATH:$PATH

esd100 · Mar 8, 2017

@yaroslavvb

Thanks for the input commands.

Some changes with that.

import tensorflow as tf
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcublas.8.0.dylib locally
I tensorflow/stream_executor/dso_loader.cc:126] Couldn't open CUDA library libcudnn.5.dylib. LD_LIBRARY_PATH: /usr/local/cuda/extras/CUPTI/lib:/usr/local/cuda/lib:/Developer/NVIDIA/CUDA-8.0/lib:/opt/intel//compilers_and_libraries_2017.0.102/mac/daal/lib:/opt/intel//compilers_and_libraries_2017.0.102/mac/ipp/lib:/opt/intel//compilers_and_libraries_2017.0.102/mac/tbb/lib:/opt/intel//compilers_and_libraries_2017.0.102/mac/tbb/lib:/opt/intel//compilers_and_libraries_2017.0.102/mac/compiler/lib:/opt/intel//compilers_and_libraries_2017.0.102/mac/mkl/lib
I tensorflow/stream_executor/cuda/cuda_dnn.cc:3517] Unable to load cuDNN DSO
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcufft.8.0.dylib locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcuda.1.dylib locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcurand.8.0.dylib locally

esd100 · Mar 8, 2017

I added

export DYLD_LIBRARY_PATH="/Developer/NVIDIA/cuDNN/lib${DYLD_LIBRARY_PATH:+:${DYLD_LIBRARY_PATH}}"

and now ...

import tensorflow as tf
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcublas.8.0.dylib locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcudnn.5.dylib locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcufft.8.0.dylib locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcuda.1.dylib locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcurand.8.0.dylib locally

esd100 · Mar 8, 2017

First trial run ...

>>> hello = tf.constant('Hello, TensorFlow!')
>>> sess = tf.Session()
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:874] OS X does not support NUMA - returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 0 with properties: 
name: GeForce GT 750M
major: 3 minor: 0 memoryClockRate (GHz) 0.9255
pciBusID 0000:01:00.0
Total memory: 2.00GiB
Free memory: 1.85GiB
I tensorflow/core/common_runtime/gpu/gpu_device.cc:906] DMA: 0 
I tensorflow/core/common_runtime/gpu/gpu_device.cc:916] 0:   Y 
I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GT 750M, pci bus id: 0000:01:00.0)
>>> print(sess.run(hello))
b'Hello, TensorFlow!'

Do you know if this output means everything is running appropriately? I'm not sure why there are all the warning about the tensorflow compiling without the special instruction sets. Any ideas?

martinwicke · Mar 8, 2017

Everything is fine. The warnings just warn you that things could be faster.

esd100 · Mar 8, 2017

@martinwicke thanks!

victorv · Mar 8, 2017

@esd100 you can suppress info and warning log messages in the shell with export TF_CPP_MIN_LOG_LEVEL=2 or in python with os.environ['TF_CPP_MIN_LOG_LEVEL']="2"

yazeedalrubyli · Mar 10, 2017

For me (MacBook Pro macOS Sierra + CUDA-8.0)

export DYLD_LIBRARY_PATH="/Developer/NVIDIA/CUDA-8.0/lib${DYLD_LIBRARY_PATH:+:${DYLD_LIBRARY_PATH}}"

Then

I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcublas.8.0.dylib locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcudnn.5.dylib locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcufft.8.0.dylib locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcuda.1.dylib locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcurand.8.0.dylib locally

It works... THX @yaroslavvb @esd100

prb12 added the installation/startup label Jun 20, 2016

prb12 assigned martinwicke Jun 20, 2016

prb12 added the triaged label Jun 20, 2016

martinwicke added the stat:awaiting response label Jun 30, 2016

aselle removed the stat:awaiting response label Jul 18, 2016

aselle removed the triaged label Jul 28, 2016

jramapuram referenced this issue Aug 10, 2016
Closed
Tensorflow Import on OSX 10.11.5 fails #3494

ghost referenced this issue Aug 15, 2016
Closed
GPU-enabled Mac build of TensorFlow version 0.10.0rc0-py2 was compiled against cuDNN v5 #3826

vinhqdang referenced this issue Sep 28, 2016
Closed
TensorFlow 0.10.0rc fails to detects CUDA device on my Macbook Pro (with nvidia 750M graphic card) #4626

yaroslavvb referenced this issue Oct 24, 2016
Closed
ipython ImportError: Library not loaded: @rpath/libcudart.8.0.dylib #5141

martinwicke closed this Nov 29, 2016

tensorflow/tensorflow

Join GitHub today

Installing Tensorflow with GPU support on os x 10.11 #2940

Comments

mts42000 commented Jun 17, 2016

Environment info

prb12 added the installation/startup label Jun 20, 2016

prb12 assigned martinwicke Jun 20, 2016

prb12 added the triaged label Jun 20, 2016

This comment has been minimized.

Velkan commented Jun 21, 2016 • edited

This comment has been minimized.

martinwicke commented Jun 23, 2016

This comment has been minimized.

Velkan commented Jun 23, 2016

This comment has been minimized.

peterswang commented Jun 23, 2016

This comment has been minimized.

Velkan commented Jun 24, 2016

This comment has been minimized.

peterswang commented Jun 24, 2016

This comment has been minimized.

tomnielsen commented Jun 29, 2016

This comment has been minimized.

yaroslavvb commented Jun 30, 2016 • edited

This comment has been minimized.

martinwicke commented Jun 30, 2016

martinwicke added the stat:awaiting response label Jun 30, 2016

This comment has been minimized.

Invisibility commented Jul 3, 2016

This comment has been minimized.

yaroslavvb commented Jul 3, 2016

This comment has been minimized.

Invisibility commented Jul 4, 2016

aselle removed the stat:awaiting response label Jul 18, 2016

This comment has been minimized.

hbfs commented Jul 26, 2016 • edited

This comment has been minimized.

bh4cyi commented Jul 26, 2016

This comment has been minimized.

hbfs commented Jul 26, 2016 • edited

This comment has been minimized.

hbfs commented Jul 26, 2016 • edited

This comment has been minimized.

yaroslavvb commented Jul 26, 2016

aselle removed the triaged label Jul 28, 2016

This comment has been minimized.

alexcolburn commented Aug 10, 2016

jramapuram referenced this issue Aug 10, 2016

Tensorflow Import on OSX 10.11.5 fails #3494

ghost referenced this issue Aug 15, 2016

GPU-enabled Mac build of TensorFlow version 0.10.0rc0-py2 was compiled against cuDNN v5 #3826

This comment has been minimized.

xuanchien commented Sep 2, 2016

This comment has been minimized.

vinhqdang commented Sep 28, 2016

vinhqdang referenced this issue Sep 28, 2016

TensorFlow 0.10.0rc fails to detects CUDA device on my Macbook Pro (with nvidia 750M graphic card) #4626

This comment has been minimized.

leocnj commented Sep 28, 2016

yaroslavvb referenced this issue Oct 24, 2016

ipython ImportError: Library not loaded: @rpath/libcudart.8.0.dylib #5141

This comment has been minimized.

martinwicke commented Nov 29, 2016

martinwicke closed this Nov 29, 2016

This comment has been minimized.

olegsinyavskiy commented Jan 15, 2017 • edited

This comment has been minimized.

kylemcdonald commented Feb 16, 2017 • edited

This comment has been minimized.

victorv commented Feb 16, 2017

This comment has been minimized.

esd100 commented Feb 20, 2017

This comment has been minimized.

hbfs commented Feb 21, 2017 • edited

Pre-requisites:

Test CUDA install

Other software installs

Install Tensorflow

Test

Velkan commented Jun 21, 2016 •

edited

yaroslavvb commented Jun 30, 2016 •

edited

hbfs commented Jul 26, 2016 •

edited

hbfs commented Jul 26, 2016 •

edited

hbfs commented Jul 26, 2016 •

edited

olegsinyavskiy commented Jan 15, 2017 •

edited

kylemcdonald commented Feb 16, 2017 •

edited

hbfs commented Feb 21, 2017 •

edited

esd100 commented Feb 22, 2017 •

edited

InstalledDir:
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

hbfs commented Feb 22, 2017 •

edited

victorv commented Feb 22, 2017 •

edited

esd100 commented Mar 2, 2017 •

edited

victorv commented Mar 3, 2017 •

edited

esd100 commented Mar 7, 2017 •

edited