New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
"gputouse" option for analyze_videos #272
Comments
Ps: I guess it's just including this line:
|
good idea (edited*: which Alex evidently already had implemented - hehe); typically I just run separate docker containers and they are individually linked to a specific GPU, but for non-Docker users, this is indeed useful. |
Yes, and you can set it e.g. by: |
Yes, for training. The same option would be good to have for the "analyze_videos" function. |
Wait, doesn't that exist as well: https://github.com/AlexEMG/DeepLabCut/blob/master/deeplabcut/pose_estimation_tensorflow/predict_videos.py#L34 |
Mhm, indeed. Then all is good and I did something silly :) |
I'm having a similar issue trying to train a network and analyze videos on 2 separate GPUs. The analyze videos function seems to call a function that utilizes both GPU's. Any ideas? Thanks! |
this is with just running analyze videos function +-----------------------------------------------------------------------------+ +-----------------------------------------------------------------------------+ |
Just pass which GPU you want to use, e.g. deeplabcut.train_network(config,gputouse=1) or deeplabcut.analyze_videos(config,videos,videotype='avi',shuffle=1,trainingsetindex=0,gputouse=0): |
We tried that too. It's just with analyze_videos. I can train 2 networks simultaneously but when I have one network running and try to analyze_videos on the second GPU I get a device CUDA:0 not supported by XLA service error When only one GPU is installed, analyze_videos will only utilize one GPU. But when the second GPU is installed, analyze_videos tries to utilize both even when I specify which GPU using gputuose so then I get the error |
Thanks for all the details. I looked into the code again and noticed that the environment variable is set after the TF session is initialized (for the predict code). I just swapped the order and updated the github repo (not pypi so far). Could you please download 2.0.6.3 and check if it works now for you? |
that worked! thank you! |
Version released: https://pypi.org/project/deeplabcut/2.0.6.3/ |
Hi there,
I have two GPUs and I could train in parallel using the "gputouse" option in train_network. I'd like to also analyze videos in parallel (as I have many) so would it be possible to include the same "gputouse" option in the analyze_videos function? If I start two different ones in parallel now I get a tensorflow error:
InternalError: Failed to create session.
Cheers,
Michael
The text was updated successfully, but these errors were encountered: