os.environ["CUDA_DEVICE_ORDER"] not work

[xiongma]

I have two GPU, I want to create 2 graph in 2 GPU, first graph in first GPU, second graph in second GPU.

1. when I create first graph in first GPU, I use os.environ["CUDA_DEVICE_ORDER"] ='0'
   2.when I create second graph in second GPU, I use os.environ["CUDA_DEVICE_ORDER"] = '1', but second graph still create on first GPU, I try many different ways, but it still not work.

It's a bug?

[xiongma]

can anyone help me, Please

[DavidWiesner]

You want specify visible devices not the device order. Use
os.environ["CUDA_VISIBLE_DEVICES"]="0" before the tensorflow import

[xiongma]

I will try, thank you

[xiongma]

@DavidWiesner this is my code, but it still not work, it occur same condition, can you help me?

[DavidWiesner]

Bert is also importing tensorflow so put your environment variables before import bert

[xiongma]

@DavidWiesner but I change my code, it still occur same condition, this is my code, after change,

[xiongma]

@DavidWiesner this is first page, it lossed

[xiongma]

this is running time log

[xiongma]

@DavidWiesner please help me, this is very important to me, I was perplex by this problem 2 days.
thank you very much!

[DavidWiesner]

It is working. In the log you see only one gpu will be used

[xiongma]

@DavidWiesner sorry, It took me so long to get back to you, But in my code, I set different gpu number, when I create class instance, I create 4 class instance, I want first instance work in fist gpu, second instance work in second gpu, third instance work in third gpu, fourth instance work in fourth gpu, but in my log, four instance still work in first gpu, I try many time, but it it same.

[xiongma]

@omalleyt12 Can you help me?

[xiongma]

@guptapriya Can you help me？

[guptapriya]

hi @policeme - have you tried creating your graph under a with tf.device('/device:GPU:1'): context?
Given you're trying to use different GPUs in the same process, that is likely a better way to specify GPUs than the environment variables. (which you can then leave to be default and let TF detect all of them).
This page has more information on how to use this API: https://www.tensorflow.org/guide/using_gpu

Hope this helps.

[xiongma]

Hi @guptapriya , but it wasn't work, this is my code and running time log, please help me.

[guptapriya]

Can you try a simple example first, such as the one from the guide and see if there you're able to get ops on different GPUs? I wonder if TF is not detecting the multiple GPUs at all for some reason. It would be easier to debug with a small simple code snippet.

[xiongma]

@guptapriya in this program, it work, I have 1 GPU, when I set GPU:1, it will occur exception, but I don't know why my first program doesn't work, can you tell me why.

[xiongma]

[xiongma]

@guptapriya this is running time log.

[guptapriya]

Do you only have 1 GPU? If yes, do you mean why your original program didn't throw exception when you tried to use other GPUs?

[xiongma]

yes

[xiongma]

if you want original code, I can put my code into github

[guptapriya]

can you check what is the value of allow_soft_placement in both the cases? if it is true, then everything will be placed on GPU:0 (when you have 1 GPU). See more details on the same page i mentioned before: https://www.tensorflow.org/guide/using_gpu
and here:
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/protobuf/config.proto#L379

[xiongma]

but when I removed allow_soft_placement in my code, I set my graph in first GPU, it appeared this exception.

[guptapriya]

That's a different error because you're trying to place the saver ops on GPU which will not work.

Also, it doesn't look like there is a bug here being reported, so I will close this ticket now. I believe stack overflow is a better venue for clarifications on usage, such as this.

[elexira]

please do not close this issue, how is this not a bug ? all these people having this issue, the least you can do is to help or leave this open.

[guptapriya]

@jaingaurav can you help look into this, and see if there is a bug, and maybe provide the recommended APIs?

[YiLing28]

@policeme Hi, Have you solved this problem? I also encountered this problem.

[xiongma]

no I haven't Cally Ma (马雄) Chongqing University of Post and Telecommunications Mobile: +86 15025700935 E-mail: mx15025700935@aliyun.com

…

------------------------------------------------------------------ 发件人：lingdavid <notifications@github.com> 发送时间：2019年9月30日(星期一) 10:55 收件人：tensorflow/tensorflow <tensorflow@noreply.github.com> 抄　送：Cally <mx15025700935@aliyun.com>; Mention <mention@noreply.github.com> 主　题：Re: [tensorflow/tensorflow] os.environ["CUDA_DEVICE_ORDER"] not work (#25152) @policeme Hi, Have you solved this problem? I also encountered this problem. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[shamangary]

I am facing the similar issue using Pytorch.
os.environ['CUDA_VISIBLE_DEVICES'] = args.gpu this somehow works on some code and some code would fail.

You may change your running command from

# this might fails with os.environ['CUDA_VISIBLE_DEVICES'] = args.gpu
python main.py

# you can use this
CUDA_VISIBLE_DEVICES="3,4,5" python main.py

However, I don't know why some codes could work some codes don't.

[htwang14]

@shamangary This solves the problem https://discuss.pytorch.org/t/why-setting-cuda-visible-devices-within-the-code-doesn-t-work/31826/2?u=ht_wang

[fengz63]

@shamangary You need to set that before the first use of cuda rather than after that. The following works well for me.