Context in use? #526

noisychannel · 2015-12-16T19:07:02Z

Running the following with GPU support :

python convolutional.py

throws the error:
F tensorflow/stream_executor/cuda/cuda_driver.cc:383] Check failed: CUDA_SUCCESS == dynload::cuCtxSetCurrent(context) (0 vs. 216)

Aborted

It seems like 216 when calling cuCtxSetCurrent (which I'm assuming assigns the context to the calling CPU thread) corresponds to CUDA_ERROR_CONTEXT_ALREADY_IN_USE.

What may be causing this error? It seems like the script successfully transfers data to the GPU and fails when initialize_all_variables() is called.

The text was updated successfully, but these errors were encountered:

noisychannel · 2015-12-16T19:11:08Z

The complete log is here : http://pastebin.com/as0fWvYv

noisychannel · 2015-12-16T19:40:18Z

And the same issue with tutorials_example_trainer. Log here : http://pastebin.com/vPkFfete

zheng-xq · 2015-12-18T20:08:01Z

@noisychannel, could you provide a bit more information about your running
environment. I see that you have two K20m on your machine. Is this a
dedicated machine, or something shared?

On Fri, Dec 18, 2015 at 11:06 AM, Derek Murray notifications@github.com
wrote:

Assigned #526 #526 to
@zheng-xq https://github.com/zheng-xq.

—
Reply to this email directly or view it on GitHub
#526 (comment).

noisychannel · 2015-12-18T21:01:20Z

Tried both scenarios :

Dedicated with 2 K20s.
Shared with 2 K20s and 1 available and selected for use.

Same issue in both cases.

digitalsword · 2016-02-13T22:58:15Z

@noisychannel I have the same issue. Did you solve it?

noisychannel · 2016-02-16T23:38:38Z

No, the issue remains.

digitalsword · 2016-02-17T19:04:34Z

@zheng-xq Is the bug fixed in the recently released tensorflow 0.7?

noisychannel · 2016-02-22T19:32:17Z

Any updates here?

zheng-xq · 2016-02-22T19:41:58Z

I've started an offline conversation with the stream-executor team, since the error originates from stream-executor. Still wait for their response.

@leary-google, @eliben, anything from the stream-executor side?

digitalsword · 2016-03-04T17:36:31Z

I am still seeing the same cuda error for tensorflow 0.7. The error is Check failed: CUDA_SUCCESS == dynload::cuCtxSetCurrent(context) (0 vs. 216)

version:

commit b88971051fbc49fa1e0b91ec1b0b60defa11697e
Merge: 5a30c8f 00986d4
Author: Derek Murray <mrry@google.com>
Date:   Fri Feb 26 05:08:35 2016 -0800

error:

++ python cifar10_train.py
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcurand.so.7.5 locally
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcufft.so.7.5 locally
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcudnn.so.4 locally
I tensorflow/stream_executor/dso_loader.cc:105] successfully opened CUDA library libcublas.so.7.5 locally
Filling queue with 20000 CIFAR images before starting to train. This will take a few minutes.
I tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 0 with properties: 
name: Tesla K40m
major: 3 minor: 5 memoryClockRate (GHz) 0.745
pciBusID 0000:08:00.0
Total memory: 11.25GiB
Free memory: 11.15GiB
I tensorflow/core/common_runtime/gpu/gpu_init.cc:126] DMA: 0 
I tensorflow/core/common_runtime/gpu/gpu_init.cc:136] 0:   Y 
I tensorflow/core/common_runtime/gpu/gpu_device.cc:718] Creating TensorFlow device (/gpu:0) -> (device: 0, name: Tesla K40m, pci bus id: 0000:08:00.0)
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:53] Creating bin of max chunk size 256B
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:53] Creating bin of max chunk size 512B
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:53] Creating bin of max chunk size 1.0KiB
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:53] Creating bin of max chunk size 2.0KiB
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:53] Creating bin of max chunk size 4.0KiB
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:53] Creating bin of max chunk size 8.0KiB
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:53] Creating bin of max chunk size 16.0KiB
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:53] Creating bin of max chunk size 32.0KiB
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:53] Creating bin of max chunk size 64.0KiB
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:53] Creating bin of max chunk size 128.0KiB
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:53] Creating bin of max chunk size 256.0KiB
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:53] Creating bin of max chunk size 512.0KiB
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:53] Creating bin of max chunk size 1.00MiB
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:53] Creating bin of max chunk size 2.00MiB
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:53] Creating bin of max chunk size 4.00MiB
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:53] Creating bin of max chunk size 8.00MiB
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:53] Creating bin of max chunk size 16.00MiB
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:53] Creating bin of max chunk size 32.00MiB
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:53] Creating bin of max chunk size 64.00MiB
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:53] Creating bin of max chunk size 128.00MiB
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:53] Creating bin of max chunk size 256.00MiB
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:107] Allocating 10.60GiB bytes.
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:118] GPU 0 memory begins at 0x13047a0000 extends to 0x15aaa4019a
F tensorflow/stream_executor/cuda/cuda_driver.cc:383] Check failed: CUDA_SUCCESS == dynload::cuCtxSetCurrent(context) (0 vs. 216)
./run.sh: line 4: 33880 Aborted                 python cifar10_train.py

rlrs · 2016-03-09T18:53:24Z

Indeed, I am seeing the same error sometimes on a shared K40. It seems to happen when someone else has completed a job, and somehow the context is not cleared? I am sure that no job is actually executing on the GPU at the time.

crscardellino · 2016-03-13T13:41:40Z

I am having the same issue with GPU:1, I can run without problems in GPU:0, but when trying to force the graph to be in GPU:1, using Graph.device(), I get the following: http://pastebin.com/ekTgqJ0U

zxvix · 2016-04-03T03:12:03Z

I encountered the error Check failed: CUDA_SUCCESS == dynload::cuCtxSetCurrent(context) (0 vs. 216) recently, which turned out to be caused by someone accidently set the compute mode of GPU to be EXCLUSIVE_THREAD. Revert it back to DEFAULT solved my error.

girving · 2016-06-06T18:43:41Z

@zheng-xq: Should we contact the Stream Executor folk offline? It looks like they might not have Github notifications turned on.

noisychannel · 2016-06-16T10:21:08Z

Issue still exists in 0.9.

noisychannel · 2016-06-16T10:24:53Z

Out of curiosity, how is this not a bigger issue? Is there a specific condition where this failure occurs? It seems like the process crashes whether all or any GPUs are available on the machine.

How do other people get around this?

danpovey · 2016-06-16T17:44:17Z

Regarding the comment of @zxvix, saying: "I encountered the error Check failed: CUDA_SUCCESS == dynload::cuCtxSetCurrent(context) (0 vs. 216) recently, which turned out to be caused by someone accidently set the compute mode of GPU to be EXCLUSIVE_THREAD. Revert it back to DEFAULT solved my error."

Sometimes on shared clusters there are valid reasons for setting exclusive mode for GPUs. Does TensorFlow require particular modes? Is EXCLUSIVE_PROCESS a possibility?

zheng-xq · 2016-06-16T17:48:17Z

Adding @henline, who is the owner of stream-executor.

kramimus · 2016-07-28T18:50:59Z

I am seeing a similar issue on 0.9 compiled from source, HEAD:

commit 554ddd9ad2d4abad5a9a31f2d245f0b1012f0d10
Merge: 89e1cc5 a0745a7
Author: yifeif <fengyifei2026@gmail.com>
Date:   Tue Jul 26 16:17:21 2016 -0700

I am on a shared cluster with a scheduler, so I should have exclusive access to the node during my time slice. It looks like exclusive mode is set, but there are no running processes at the time time I try to use it:

[2016-07-28T17:59:19Z]: +------------------------------------------------------+                       
[2016-07-28T17:59:19Z]: | NVIDIA-SMI 352.39     Driver Version: 352.39         |                       
[2016-07-28T17:59:19Z]: |-------------------------------+----------------------+----------------------+
[2016-07-28T17:59:19Z]: | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
[2016-07-28T17:59:19Z]: | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
[2016-07-28T17:59:19Z]: |===============================+======================+======================|
[2016-07-28T17:59:19Z]: |   0  Tesla K40m          Off  | 0000:08:00.0     Off |                    0 |
[2016-07-28T17:59:19Z]: | N/A   25C    P8    19W / 235W |     23MiB / 11519MiB |      0%    E. Thread |
[2016-07-28T17:59:19Z]: +-------------------------------+----------------------+----------------------+
[2016-07-28T17:59:19Z]:                                                                                
[2016-07-28T17:59:19Z]: +-----------------------------------------------------------------------------+
[2016-07-28T17:59:19Z]: | Processes:                                                       GPU Memory |
[2016-07-28T17:59:19Z]: |  GPU       PID  Type  Process name                               Usage      |
[2016-07-28T17:59:19Z]: |=============================================================================|
[2016-07-28T17:59:19Z]: |  No running processes found                                                 |
[2016-07-28T17:59:19Z]: +-----------------------------------------------------------------------------+
[2016-07-28T17:59:21Z]: Using TensorFlow backend.
[2016-07-28T17:59:22Z]: I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcurand.so.7.5 locally
[2016-07-28T17:59:22Z]: I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcuda.so.1 locally
[2016-07-28T17:59:22Z]: I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcufft.so.7.5 locally
[2016-07-28T17:59:23Z]: I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcudnn.so.4.0.7 locally
[2016-07-28T17:59:23Z]: I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcublas.so.7.5 locally
[2016-07-28T17:59:34Z]: I tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 0 with properties: 
[2016-07-28T17:59:34Z]: name: Tesla K40m
[2016-07-28T17:59:34Z]: major: 3 minor: 5 memoryClockRate (GHz) 0.745
[2016-07-28T17:59:34Z]: pciBusID 0000:08:00.0
[2016-07-28T17:59:34Z]: Total memory: 11.25GiB
[2016-07-28T17:59:34Z]: Free memory: 11.15GiB
[2016-07-28T17:59:34Z]: I tensorflow/core/common_runtime/gpu/gpu_init.cc:126] DMA: 0 
[2016-07-28T17:59:34Z]: I tensorflow/core/common_runtime/gpu/gpu_init.cc:136] 0:   Y 
[2016-07-28T17:59:34Z]: I tensorflow/core/common_runtime/gpu/gpu_device.cc:839] Creating TensorFlow device (/gpu:0) -> (device: 0, name: Tesla K40m, pci bus id: 0000:08:00.0)
[2016-07-28T17:59:34Z]: F tensorflow/stream_executor/cuda/cuda_driver.cc:395] Check failed: CUDA_SUCCESS == dynload::cuCtxSetCurrent(cuda_context->context()) (0 vs. 216)
[2016-07-28T17:59:34Z]: X_train shape: (60000, 1, 28, 28)
[2016-07-28T17:59:34Z]: 60000 train samples
[2016-07-28T17:59:34Z]: 10000 test samples
[2016-07-28T17:59:41Z]: /tmp/wrapper5271742824235601482.sh: line 12: 61655 Aborted                 (core dumped) python mnist_cnn.py
[2016-07-28T17:59:41Z]: Exited with code 0

danpovey · 2016-07-28T19:08:59Z

Tensorflow guys, if I were you I would change the assert statement that
fails here:

cuda/cuda_driver.cc:395] Check failed: CUDA_SUCCESS ==
dynload::cuCtxSetCurrent(cuda_context->context()) (0 vs. 216)

to some code that prints out the text form of the CUDA exit code, and maybe
for good measure tries to invoke nvidia-smi to get extra information. We
found this necessary in Kaldi in order to ensure that when there are
problems, all the information needed is in the log.
Dan

On Thu, Jul 28, 2016 at 11:52 AM, Mark Whitney notifications@github.com
wrote:

I am seeing a similar issue on 0.9 compiled from source, HEAD:

commit 554ddd9
Merge: 89e1cc5 a0745a7
Author: yifeif fengyifei2026@gmail.com
Date: Tue Jul 26 16:17:21 2016 -0700

I am on a shared cluster with a scheduler, so I should have exclusive
access to the node during my time slice. It looks like exclusive mode is
set, but there are no running processes at the time time I try to use it:

2016-07-28T17:59:19Z: | NVIDIA-SMI 352.39 Driver Version: 352.39 |
2016-07-28T17:59:19Z: |-------------------------------+----------------------+----------------------+
2016-07-28T17:59:19Z: | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
2016-07-28T17:59:19Z: | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
2016-07-28T17:59:19Z: |===============================+======================+======================|
2016-07-28T17:59:19Z: | 0 Tesla K40m Off | 0000:08:00.0 Off | 0 |
2016-07-28T17:59:19Z: | N/A 25C P8 19W / 235W | 23MiB / 11519MiB | 0% E. Thread |
2016-07-28T17:59:19Z: +-------------------------------+----------------------+----------------------+
2016-07-28T17:59:19Z:
2016-07-28T17:59:19Z: +-----------------------------------------------------------------------------+
2016-07-28T17:59:19Z: | Processes: GPU Memory |
2016-07-28T17:59:19Z: | GPU PID Type Process name Usage |
2016-07-28T17:59:19Z: |=============================================================================|
2016-07-28T17:59:19Z: | No running processes found |
2016-07-28T17:59:19Z: +-----------------------------------------------------------------------------+
[2016-07-28T17:59:21Z]: Using TensorFlow backend.
[2016-07-28T17:59:22Z]: I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcurand.so.7.5 locally
[2016-07-28T17:59:22Z]: I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcuda.so.1 locally
[2016-07-28T17:59:22Z]: I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcufft.so.7.5 locally
[2016-07-28T17:59:23Z]: I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcudnn.so.4.0.7 locally
[2016-07-28T17:59:23Z]: I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcublas.so.7.5 locally
[2016-07-28T17:59:34Z]: I tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 0 with properties:
[2016-07-28T17:59:34Z]: name: Tesla K40m
[2016-07-28T17:59:34Z]: major: 3 minor: 5 memoryClockRate (GHz) 0.745
[2016-07-28T17:59:34Z]: pciBusID 0000:08:00.0
[2016-07-28T17:59:34Z]: Total memory: 11.25GiB
[2016-07-28T17:59:34Z]: Free memory: 11.15GiB
[2016-07-28T17:59:34Z]: I tensorflow/core/common_runtime/gpu/gpu_init.cc:126] DMA: 0
[2016-07-28T17:59:34Z]: I tensorflow/core/common_runtime/gpu/gpu_init.cc:136] 0: Y
[2016-07-28T17:59:34Z]: I tensorflow/core/common_runtime/gpu/gpu_device.cc:839] Creating TensorFlow device (/gpu:0) -> (device: 0, name: Tesla K40m, pci bus id: 0000:08:00.0)
[2016-07-28T17:59:34Z]: F tensorflow/stream_executor/cuda/cuda_driver.cc:395] Check failed: CUDA_SUCCESS == dynload::cuCtxSetCurrent(cuda_context->context()) (0 vs. 216)
[2016-07-28T17:59:34Z]: X_train shape: (60000, 1, 28, 28)
[2016-07-28T17:59:34Z]: 60000 train samples
[2016-07-28T17:59:34Z]: 10000 test samples
[2016-07-28T17:59:41Z]: /tmp/wrapper5271742824235601482.sh: line 12: 61655 Aborted (core dumped) python mnist_cnn.py
[2016-07-28T17:59:41Z]: Exited with code 0

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#526 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADJVu5zDQGnFRHmS-sFWKkA2dnlN3v1Cks5qaPqBgaJpZM4G2yCN
.

henline · 2016-07-29T18:36:17Z

Hi, I'm the owner of StreamExecutor. Sorry for arriving late to this discussion.

I believe this problem is caused in all cases by GPUs with their compute mode set to EXCLUSIVE_THREAD (just as mentioned by @zxvix). The solution is to set the compute mode to DEFAULT or EXCLUSIVE_PROCESS, which can be done via one of the following commands:

$ nvidia-smi --compute-mode=0 # for DEFAULT
$ nvidia-smi --compute-mode=3 # for EXCLUSIVE_PROCESS

The nvidia-smi -q command can also be used to query the current compute mode of the device.

If anyone is seeing this error when the device compute mode is either DEFAULT or EXCLUSIVE_PROCESS, please let me know, because I don't think that should be possible.

StreamExecutor will not work in either EXCLUSIVE_THREAD or PROHIBITED compute mode, but in response to @danpovey's question about shared clusters, EXCLUSIVE_PROCESS mode should be fine.

There are no plans in StreamExecutor to support EXCLUSIVE_THREAD mode because it is listed as deprecated in the nvidia-smi help message. There are also no plans to support PROHIBITED mode because I think that mode prevents the creation of contexts, and StreamExecutor cannot function with that restriction.

In response to @danpovey's suggestion about adding a better error message for this case, I think that's a good idea. I will work on getting a patch up to warn about the device compute mode if cuCtxSetCurrent fails and the compute mode is set to an unsupported setting.

kramimus · 2016-07-29T19:46:15Z

Thanks, good to know it should work in EXCLUSIVE_PROCESS.

alextp · 2016-08-15T22:58:25Z

So it sounds like this is working as intended, since StreamExecutor doesn't plan on supporting modes other than EXCLUSIVE_PROCESS.

noisychannel · 2016-08-16T15:45:43Z

Just to confirm, StreamExecutor works with EXCLUSIVE_PROCESS. Hopefully, @danpovey's suggestion about better error messages will be added in soon. It may be hard for people to search for this issue.

MidoAssran · 2017-03-16T02:20:42Z

For anyone using GPU based tensorflow on Compute Canada resources, submitting the job by specifying EXCLUSIVE_PROCESS worked for me.

zafarali · 2017-04-17T15:50:17Z

@MidoAssran Thanks for that! Will try it out.

201power · 2017-10-03T00:20:02Z

I meet this error

change compute_mode does not solve the issue, any guidance?

danpovey · 2017-10-03T00:30:31Z

I can't really see what it's saying (next time paste as text), but in any case I don't know what this might be. Maybe all the GPUs were already in use.

…

On Mon, Oct 2, 2017 at 8:22 PM, 201power ***@***.***> wrote: I meet this error [image: image] <https://user-images.githubusercontent.com/7299296/31105041-dfe35b10-a795-11e7-9d64-22c460ebaefd.png> change compute_mode does not solve the issue, any guidance? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#526 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADJVux59GbGOM5GlqUBTsst_i5N6RwA_ks5soX42gaJpZM4G2yCN> .

201power · 2017-10-04T16:33:14Z

The GPU might be in use. This happened in a TF session followed immediately by another TF session.
I am not sure how to tell that if GPU is in use, since TF won't release memory after the first session finished.

dee6600 · 2018-02-27T12:26:58Z

changing compute mode didn't solve the problem, i am getting the same error, my cards are two tesla k80 's
and driver version 375 cuda version 8.0 tensorflow version 1.1

tumusudheer · 2019-01-26T09:25:41Z

I'm getting the same error (or similar one)
2019-01-26 01:22:06.728780: F tensorflow/stream_executor/cuda/cuda_driver.cc:206] Check failed: CUDA_SUCCESS == cuCtxSetCurrent(cuda_context->context()) (0 vs. 4) Aborted (core dumped)
Sample of my Nvidia-smi command output:
`Sat Jan 26 01:22:29 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.48 Driver Version: 410.48 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce RTX 2080 Off | 00000000:17:00.0 Off | N/A |
| 0% 42C P8 1W / 225W | 0MiB / 7952MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 GeForce RTX 2080 Off | 00000000:65:00.0 On | N/A |
| 0% 47C P8 13W / 225W | 166MiB / 7951MiB | 6% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 1 1101 G /usr/lib/xorg/Xorg 165MiB |
+-----------------------------------------------------------------------------+
`

I'm running TF verison 1.12 on Ubuntu 18.04 Cuda 10.0, CUDNN 7.3.1

zhouchaopku · 2020-01-15T09:33:32Z

Hi, I'm the owner of StreamExecutor. Sorry for arriving late to this discussion.

I believe this problem is caused in all cases by GPUs with their compute mode set to EXCLUSIVE_THREAD (just as mentioned by @zxvix). The solution is to set the compute mode to DEFAULT or EXCLUSIVE_PROCESS, which can be done via one of the following commands:
$ nvidia-smi --compute-mode=0 # for DEFAULT
$ nvidia-smi --compute-mode=3 # for EXCLUSIVE_PROCESS
The nvidia-smi -q command can also be used to query the current compute mode of the device.

If anyone is seeing this error when the device compute mode is either DEFAULT or EXCLUSIVE_PROCESS, please let me know, because I don't think that should be possible.

StreamExecutor will not work in either EXCLUSIVE_THREAD or PROHIBITED compute mode, but in response to @danpovey's question about shared clusters, EXCLUSIVE_PROCESS mode should be fine.

There are no plans in StreamExecutor to support EXCLUSIVE_THREAD mode because it is listed as deprecated in the nvidia-smi help message. There are also no plans to support PROHIBITED mode because I think that mode prevents the creation of contexts, and StreamExecutor cannot function with that restriction.

In response to @danpovey's suggestion about adding a better error message for this case, I think that's a good idea. I will work on getting a patch up to warn about the device compute mode if cuCtxSetCurrent fails and the compute mode is set to an unsupported setting.

@henline Sorry to trouble you. I have set the mode to DEFAULT (nvidia-smi --compute-mode=0), and it indeed takes effect by the info. from nvidia-smi -q. But I still got the error "tensorflow/stream_executor/cuda/cuda_driver.cc:225] Check failed: CUDA_SUCCESS == cuCtxSetCurrent(cuda_context->context()) (0 vs. 3)".

…int8_pooling Disable qint8 forward pooling on ROCm properly.

mrry assigned zheng-xq Dec 18, 2015

vrv mentioned this issue Feb 14, 2016

cuda error on K40m GPU #1088

Closed

girving added the triaged label Jun 6, 2016

zheng-xq assigned zheng-xq and unassigned zheng-xq Jun 16, 2016

aselle removed the triaged label Jul 28, 2016

alextp closed this as completed Aug 15, 2016

tumusudheer unassigned zheng-xq Jan 26, 2019

tumusudheer mentioned this issue Jan 27, 2019

Check failed: CUDA_SUCCESS == cuCtxSetCurrent(cuda_context->context()) (0 vs. 4) #25220

Closed

darkbuck pushed a commit to darkbuck/tensorflow that referenced this issue Jan 23, 2020

Merge pull request tensorflow#526 from ROCmSoftwarePlatform/disable_q…

3784582

…int8_pooling Disable qint8 forward pooling on ROCm properly.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Context in use? #526

Context in use? #526

noisychannel commented Dec 16, 2015

noisychannel commented Dec 16, 2015

noisychannel commented Dec 16, 2015

zheng-xq commented Dec 18, 2015

noisychannel commented Dec 18, 2015

digitalsword commented Feb 13, 2016

noisychannel commented Feb 16, 2016

digitalsword commented Feb 17, 2016

noisychannel commented Feb 22, 2016

zheng-xq commented Feb 22, 2016

digitalsword commented Mar 4, 2016

rlrs commented Mar 9, 2016

crscardellino commented Mar 13, 2016

zxvix commented Apr 3, 2016

girving commented Jun 6, 2016

noisychannel commented Jun 16, 2016

noisychannel commented Jun 16, 2016

danpovey commented Jun 16, 2016

zheng-xq commented Jun 16, 2016

kramimus commented Jul 28, 2016

danpovey commented Jul 28, 2016

henline commented Jul 29, 2016

kramimus commented Jul 29, 2016

alextp commented Aug 15, 2016

noisychannel commented Aug 16, 2016

MidoAssran commented Mar 16, 2017

zafarali commented Apr 17, 2017

201power commented Oct 3, 2017

danpovey commented Oct 3, 2017 via email

201power commented Oct 4, 2017 •

edited

Loading

dee6600 commented Feb 27, 2018 •

edited

Loading

tumusudheer commented Jan 26, 2019

zhouchaopku commented Jan 15, 2020 •

edited

Loading

Context in use? #526

Context in use? #526

Comments

noisychannel commented Dec 16, 2015

noisychannel commented Dec 16, 2015

noisychannel commented Dec 16, 2015

zheng-xq commented Dec 18, 2015

noisychannel commented Dec 18, 2015

digitalsword commented Feb 13, 2016

noisychannel commented Feb 16, 2016

digitalsword commented Feb 17, 2016

noisychannel commented Feb 22, 2016

zheng-xq commented Feb 22, 2016

digitalsword commented Mar 4, 2016

rlrs commented Mar 9, 2016

crscardellino commented Mar 13, 2016

zxvix commented Apr 3, 2016

girving commented Jun 6, 2016

noisychannel commented Jun 16, 2016

noisychannel commented Jun 16, 2016

danpovey commented Jun 16, 2016

zheng-xq commented Jun 16, 2016

kramimus commented Jul 28, 2016

danpovey commented Jul 28, 2016

henline commented Jul 29, 2016

kramimus commented Jul 29, 2016

alextp commented Aug 15, 2016

noisychannel commented Aug 16, 2016

MidoAssran commented Mar 16, 2017

zafarali commented Apr 17, 2017

201power commented Oct 3, 2017

danpovey commented Oct 3, 2017 via email

201power commented Oct 4, 2017 • edited Loading

dee6600 commented Feb 27, 2018 • edited Loading

tumusudheer commented Jan 26, 2019

zhouchaopku commented Jan 15, 2020 • edited Loading

201power commented Oct 4, 2017 •

edited

Loading

dee6600 commented Feb 27, 2018 •

edited

Loading

zhouchaopku commented Jan 15, 2020 •

edited

Loading