Please go to Stack Overflow for help and support:

https://stackoverflow.com/questions/tagged/tensorflow

If you open a GitHub issue, here is our policy:

1. It must be a bug, a feature request, or a significant problem with documentation (for small docs fixes please send a PR instead).
2. The form below must be filled out.
3. It shouldn't be a TensorBoard issue. Those go here.

Here's why we have that policy: TensorFlow developers respond to issues. We want to focus on work that benefits the whole community, e.g., fixing bugs and adding features. Support only helps individuals. GitHub also notifies thousands of people when issues are filed. We want them to see you communicating an interesting problem, rather than being redirected to Stack Overflow.

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow): YES

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 16.04

• TensorFlow installed from (source or binary): binary

• TensorFlow version (use command below): v1.8.0-0-g93bc2e2072 1.8.0
  I have also tried this on 16.0 and 17.0, it crashes both of them.
  13.0 and 15.0 are fine.

• Python version: 3.6.3

• Bazel version (if compiling from source):

• GCC/Compiler version (if compiling from source):

• CUDA/cuDNN version: Both 8.0, and 9.1 (with 9.0 libraries)

• GPU model and memory: GeForce GTX 1080 Ti. with 11178 MiB

• Exact command to reproduce:

import tensorflow as tf
G =tf.Graph()
sess1 = tf.Session(graph=G, config=tf.ConfigProto(log_device_placement=False,gpu_options=tf.GPUOptions(allow_growth=True,visible_device_list='0')))
sess2 = tf.Session(graph=G, config=tf.ConfigProto(log_device_placement=False,gpu_options=tf.GPUOptions(allow_growth=True,visible_device_list='1')))

Running the second tf.Session command crashes with the following error:

F tensorflow/core/common_runtime/gpu/gpu_id_manager.cc:45] Check failed: cuda_gpu_id.value() == result.first->second (1 vs. 0)Mapping the same TfGpuId to a different CUDA GPU id. TfGpuId: 0 Existing mapped CUDA GPU id: 0 CUDA GPU id being tried to map to: 1

You can collect some of this information using our environment capture script:

https://github.com/tensorflow/tensorflow/tree/master/tools/tf_env_collect.sh

You can obtain the TensorFlow version with

python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"

Describe the problem

Describe the problem clearly here. Be sure to convey here why it's a bug in TensorFlow or a feature request.

The GPU remapping using visible_device_list is broken. This works fine in Tensorflow 1.3 and 1.5, but is completely broken (crashes the program) in 1.6, 1.7 and 1.8.
As far as I can tell from reading tensorflow/include/tensorflow/core/common_runtime/gpu/gpu_id.h
this mechanism is supposed to still work the same way it used to.

Source code / logs

Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem.

import tensorflow as tf
G =tf.Graph()
sess1 = tf.Session(graph=G, config=tf.ConfigProto(log_device_placement=False,gpu_options=tf.GPUOptions(allow_growth=True,visible_device_list='0')))
sess2 = tf.Session(graph=G, config=tf.ConfigProto(log_device_placement=False,gpu_options=tf.GPUOptions(allow_growth=True,visible_device_list='1')))

F tensorflow/core/common_runtime/gpu/gpu_id_manager.cc:45] Check failed: cuda_gpu_id.value() == result.first->second (1 vs. 0)Mapping the same TfGpuId to a different CUDA GPU id. TfGpuId: 0 Existing mapped CUDA GPU id: 0 CUDA GPU id being tried to map to: 1