Tensorflow Failed to create Session

[sachinprabhu007]

Hi

I tried basic program in python shell. It fails to create session. Please assist.

Thanks

Python 2.7.6 (default, Oct 26 2016, 20:30:19)
[GCC 4.8.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcublas.so.7.5 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcudnn.so.5 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcufft.so.7.5 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcurand.so.7.5 locally
>>> hello = tf.constant('hi,tensorflow')
>>> sess = tf.Session()
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:910] successful NUMA node                                                                                                              read from SysFS had negative value (-1), but there must be at least one NUMA no                                                                                                             de, so returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 0 with prop                                                                                                             erties:
name: Tesla K40c
major: 3 minor: 5 memoryClockRate (GHz) 0.745
pciBusID 0000:02:00.0
Total memory: 11.17GiB
Free memory: 11.10GiB
W tensorflow/stream_executor/cuda/cuda_driver.cc:590] creating context when one                                                                                                              is currently active; existing: 0x2e0fe90
E tensorflow/core/common_runtime/direct_session.cc:137] Internal: failed initial                                                                                                             izing StreamExecutor for CUDA device ordinal 1: Internal: failed call to cuDevic                                                                                                             ePrimaryCtxRetain: CUDA_ERROR_INVALID_DEVICE
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.                                                                                                             py", line 1187, in __init__
    super(Session, self).__init__(target, graph, config=config)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.                                                                                                             py", line 552, in __init__
    self._session = tf_session.TF_NewDeprecatedSession(opts, status)
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/error                                                                                                             s_impl.py", line 469, in raise_exception_on_not_ok_status
    pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.InternalError: Failed to create session.

[yaroslavvb]

maybe out of GPU memory? Try running with export CUDA_VISIBLE_DEVICES=''

[aselle]

Please provide details about what platform you are using (operating system, architecture). Also include your TensorFlow version. Also, did you compile from source or install a binary? Make sure you also include the exact command if possible to produce the output included in your test case. If you are unclear what to include see the issue template displayed in the Github new issue template.

We ask for this in the issue submission template, because it is really difficult to help without that information. Thanks!

[sachinprabhu007]

System details : ubuntu 14.04 x86_64
Tensorflow version: 0.10.0
Cuda Version : 7.5

I was able to install cpu version and execute my model but GPU version fails. I tried using latest Tensorflow Version 1.0 but again it was not able to create session.

[aselle]

Try upgrading your cuda drivers and sdk?

[girving]

Automatically closing due to lack of recent activity. Please update the issue when new information becomes available, and we will reopen the issue. Thanks!

[khushhallchandra]

Restarting the system helped in my case.

[sumedhvdatar]

Getting unable to create tensorflow session. Error occurs even after restarting the system. Also I checked that the GPU consumption is almost 0.

i am using windows operating system. Let me know if you need more information

[PhilippeNguyen]

Updating NVIDIA drivers fixed this for me

[Zhang-Wenwen]

Can you provide the use information of your GPUS? Maybe somebody else is using the GPU, which leads to this problem. This is the case for me .

[Zhongan-Wang]

I have met this problem ,and i didn't solve it. i use anonaconda3 commond conda install tensorflow-gpu but when i run a python file ,it told me tensorflow.python.framework.errors_impl.InternalError: Failed to create session.I don't think out of my memory.Because I have eight NVIDIA K80

[baker-travis]

For me, I had to downgrade to tensorflow 1.9.0. It might be a cuda toolkit incompatibility issue, I'm not sure. It looked like tensorflow v10 used cuda toolkit 9.2, while I only had 9.0 installed.

[Cynthia0629]

That helped me a lot when I updated my tensorflow to 1.10.0, thank you very much @baker-travis

[nateGeorge]

In the case I just solved, it was updating the GPU driver to the latest and installing the cuda toolkit. First, the ppa was added and GPU driver installed:

sudo add-apt-repository ppa:graphics-drivers/ppa
sudo apt update
sudo apt install nvidia-390

After adding the ppa, it showed options for driver versions, and 390 was the latest 'stable' version that was shown.

Then install the cuda toolkit:

sudo apt install nvidia-cuda-toolkit

Then reboot:

sudo reboot

It updated the drivers to a newer version than the 390 originally installed in the first step (it was 410; this was a p2.xlarge instance on AWS).

[rudy-letote]

for me i key "conda install cudnn==7.1.2“ 。
my previous cudnn edition was 7.3.1 which respondes to cuda9.2_0 ,but my cuda edition is 9.0.175.so i reduce the edition of cudnn to 7.1.2 which respondes to cuda 9.0.0.

[ShilpaSangappa]

Adding these lines worked for me.

#Provide the GPU number to be used
import os
os.environ['CUDA_VISIBLE_DEVICES'] =''

[DataStunner]

If your GPU utility shows 0% and still tensorflow failed to create the session, there are some forcefully closed tensorflow session. These will not usally visible with nvidia-smi. But sudo fuser -v /dev/nvidia* can capture these processes. Killing those processes can help without restarting the server.