Context in use?

[noisychannel]

Running the following with GPU support :

python convolutional.py

throws the error:
F tensorflow/stream_executor/cuda/cuda_driver.cc:383] Check failed: CUDA_SUCCESS == dynload::cuCtxSetCurrent(context) (0 vs. 216)

Aborted

It seems like 216 when calling cuCtxSetCurrent (which I'm assuming assigns the context to the calling CPU thread) corresponds to CUDA_ERROR_CONTEXT_ALREADY_IN_USE.

What may be causing this error? It seems like the script successfully transfers data to the GPU and fails when initialize_all_variables() is called.

[noisychannel]

The complete log is here : http://pastebin.com/as0fWvYv

[noisychannel]

And the same issue with tutorials_example_trainer. Log here : http://pastebin.com/vPkFfete

[zheng-xq]

@noisychannel, could you provide a bit more information about your running
environment. I see that you have two K20m on your machine. Is this a
dedicated machine, or something shared?

On Fri, Dec 18, 2015 at 11:06 AM, Derek Murray notifications@github.com
wrote:

Assigned #526 #526 to
@zheng-xq https://github.com/zheng-xq.

—
Reply to this email directly or view it on GitHub
#526 (comment).

[noisychannel]

Tried both scenarios :

1. Dedicated with 2 K20s.
2. Shared with 2 K20s and 1 available and selected for use.

Same issue in both cases.

[digitalsword]

@noisychannel I have the same issue. Did you solve it?

[noisychannel]

No, the issue remains.

[digitalsword]

@zheng-xq Is the bug fixed in the recently released tensorflow 0.7?

[noisychannel]

Any updates here?

[zheng-xq]

I've started an offline conversation with the stream-executor team, since the error originates from stream-executor. Still wait for their response.

@leary-google, @eliben, anything from the stream-executor side?

[digitalsword]

I am still seeing the same cuda error for tensorflow 0.7. The error is Check failed: CUDA_SUCCESS == dynload::cuCtxSetCurrent(context) (0 vs. 216)

version:

commit b88971051fbc49fa1e0b91ec1b0b60defa11697e
Merge: 5a30c8f 00986d4
Author: Derek Murray <mrry@google.com>
Date:   Fri Feb 26 05:08:35 2016 -0800

error:

++ python cifar10_train.py
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcurand.so.7.5 locally
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcufft.so.7.5 locally
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcudnn.so.4 locally
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcublas.so.7.5 locally
Filling queue with 20000 CIFAR images before starting to train. This will take a few minutes.
I tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 0 with properties: 
name: Tesla K40m
major: 3 minor: 5 memoryClockRate (GHz) 0.745
pciBusID 0000:08:00.0
Total memory: 11.25GiB
Free memory: 11.15GiB
I tensorflow/core/common_runtime/gpu/gpu_init.cc:126] DMA: 0 
I tensorflow/core/common_runtime/gpu/gpu_init.cc:136] 0:   Y 
I tensorflow/core/common_runtime/gpu/gpu_device.cc:718] Creating TensorFlow device (/gpu:0) -> (device: 0, name: Tesla K40m, pci bus id: 0000:08:00.0)
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:53] Creating bin of max chunk size 256B
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:53] Creating bin of max chunk size 512B
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:53] Creating bin of max chunk size 1.0KiB
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:53] Creating bin of max chunk size 2.0KiB
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:53] Creating bin of max chunk size 4.0KiB
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:53] Creating bin of max chunk size 8.0KiB
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:53] Creating bin of max chunk size 16.0KiB
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:53] Creating bin of max chunk size 32.0KiB
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:53] Creating bin of max chunk size 64.0KiB
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:53] Creating bin of max chunk size 128.0KiB
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:53] Creating bin of max chunk size 256.0KiB
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:53] Creating bin of max chunk size 512.0KiB
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:53] Creating bin of max chunk size 1.00MiB
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:53] Creating bin of max chunk size 2.00MiB
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:53] Creating bin of max chunk size 4.00MiB
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:53] Creating bin of max chunk size 8.00MiB
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:53] Creating bin of max chunk size 16.00MiB
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:53] Creating bin of max chunk size 32.00MiB
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:53] Creating bin of max chunk size 64.00MiB
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:53] Creating bin of max chunk size 128.00MiB
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:53] Creating bin of max chunk size 256.00MiB
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:107] Allocating 10.60GiB bytes.
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:118] GPU 0 memory begins at 0x13047a0000 extends to 0x15aaa4019a
F tensorflow/stream_executor/cuda/cuda_driver.cc:383] Check failed: CUDA_SUCCESS == dynload::cuCtxSetCurrent(context) (0 vs. 216)
./run.sh: line 4: 33880 Aborted                 python cifar10_train.py

[rlrs]

Indeed, I am seeing the same error sometimes on a shared K40. It seems to happen when someone else has completed a job, and somehow the context is not cleared? I am sure that no job is actually executing on the GPU at the time.

[crscardellino]

I am having the same issue with GPU:1, I can run without problems in GPU:0, but when trying to force the graph to be in GPU:1, using Graph.device(), I get the following: http://pastebin.com/ekTgqJ0U

[zxvix]

I encountered the error Check failed: CUDA_SUCCESS == dynload::cuCtxSetCurrent(context) (0 vs. 216) recently, which turned out to be caused by someone accidently set the compute mode of GPU to be EXCLUSIVE_THREAD. Revert it back to DEFAULT solved my error.