New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

volatile gpu-util=0% when training #83

Open

actor1996 opened this issue Nov 20, 2021 · 1 comment

actor1996 commented Nov 20, 2021 •

edited

here are the settings in https://github.com/HRNet/HRNet-Facial-Landmark-Detection/blob/master/experiments/wflw/face_alignment_wflw_hrnet_w18.yaml

GPUS: (0,1,2,)
WORKERS: 33
PRINT_FREQ: 3
...

I didn't change other options

as showed in the picture above, the gpus 0,1,2 I used for train hrnet got 0% usage

How could that happen, is it because too much time is spent on processing data with the cpu

tianmaxingkong12 commented Apr 21, 2022

Preprocessing takes too much time. It's just a snap by using nvidia-smi. when using watch , it 's not always zero.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment