Object detection: when training, GPU utilization started high, and became low/idle later on

[mrxiaohe]

System information

• What is the top-level directory of the model you are using: \Tensorflow\models\research\object_detection
• Have I written custom code: No
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows Server 2016
• TensorFlow installed from (source or binary): binary
• TensorFlow version (use command below): 1.13.1
• Python version: 3.7.3
• CUDA/cuDNN version: 10/7.1
• GPU model and memory: Tesla V100 - 16 GB
• Bazel version: N/A
• Exact command to reproduce: N/A

Describe the current behavior
I am trying to train a face detector by using the Object Detection API. Specifically, I am using 4000 images from the Wider Faces data set for training/validation, and using faster_rcnn_resnet101_coco for fine-tuning. I split the train set images into 5 shards, and in the .config file, I changed the min_dimension and max_dimension to 300 and 512, respectively, in hopes of faster training.

After I kicked off training, I noticed that the GPU memory usage was about 95%. GPU volatile-utilization and CPU utilization on the other hand were behaving oddly. Specifically, during the very first ~3k steps, GPU volatile-utilization was between 60% and 95%, and the CPU utilization was only around 8%. Global_step/sec essentially plateaued around 8 steps/sec within the first 3k steps.

From ~3k steps and onward, CPU utilization increased to 80-100% whereas GPU volatile utilization oscillated between 0% to 30% and for the most part, it was idle (0%). GPU memory usage continued to be high.

Starting from ~4k steps. global_step/sec dropped to around 1 step/sec (as shown below)

Unlike training, inference does seem to make use of GPU -- GPU volatile utilization is between 70%-100% during inference.

Describe the expected behavior
Given the initial high GPU volatile utilization rate, I was expecting the high util-rate to persist, so it was a bit strange to see that after the initial 3k steps, the training process only occasionally used the GPU, and mostly relied on the CPU.

Any pointers/help would be much appreciated. Thanks!

[tensorflowbutler]

Thank you for your post. We noticed you have not filled out the following field in the issue template. Could you update them if they are relevant in your case, or leave them as N/A? Thanks.
What is the top-level directory of the model you are using
Bazel version
Exact command to reproduce

[mrxiaohe]

System information

• What is the top-level directory of the model you are using: \Tensorflow\models\research\object_detection
• Have I written custom code: No
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows Server 2016
• TensorFlow installed from (source or binary): binary
• TensorFlow version (use command below): 1.13.1
• Python version: 3.7.3
• CUDA/cuDNN version: 10/7.1
• GPU model and memory: Tesla V100 - 16 GB
• Bazel version: N/A
• Exact command to reproduce: N/A

[hukkelas]

I'm struggling with the same issue. Did you find any solution to this?

[rhossei2]

I recently ran into this same issue and realized it was due to long evaluation cycles. I was using the default parameters that sampled my entire validation dataset EVERY 5 minutes. Since my validation dataset was large, each cycle took over 15 minutes. This was causing constant evaluations as each cycle overlapped the next. The solution is to pass sample_1_of_n_eval_examples to your training script and increase its value from the default 1 to something like 3. Another thing to try is to increase the evaluation intervals (eval_interval_secs) to something higher than the default 5 minutes.

(btw my setup is very similar as I'm running Tensorflow 1.13.1 on Tesla V100 with Cuda 10)

[suwoongleekor]

I am struggling with exactly the same issue, and from lots of google search, rhossei2's two solutions could be possible solutions.

However, I tried the above two solutions, but there are some problems. In detail,
Sol1: larger sample_1_of_n_eval_examples) It effectively reduces system memory usage and maintains training speed. However, its evaluation results are quite different from the real case. In my case, sample_1_of_n_eval_examples=20 and the AP difference is about 2~3%.

Sol2: increase eval_interval_secs) I added this option to pipeline config, but it seems that this parameter is not working, as other people reported. #5801, #5144, ...

Does anyone solve this issue clearly?
In this moment, train with solution 1, and evaluate some ckpts manually would be best.

I recently ran into this same issue and realized it was due to long evaluation cycles. I was using the default parameters that sampled my entire validation dataset EVERY 5 minutes. Since my validation dataset was large, each cycle took over 15 minutes. This was causing constant evaluations as each cycle overlapped the next. The solution is to pass sample_1_of_n_eval_examples to your training script and increase its value from the default 1 to something like 3. Another thing to try is to increase the evaluation intervals (eval_interval_secs) to something higher than the default 5 minutes.

(btw my setup is very similar as I'm running Tensorflow 1.13.1 on Tesla V100 with Cuda 10)

[tensorflowbutler]

Hi There,
We are checking to see if you still need help on this, as this seems to be an old issue. Please update this issue with the latest information, code snippet to reproduce your issue and error you are seeing.
If we don't hear from you in the next 7 days, this issue will be closed automatically. If you don't need help on this issue any more, please consider closing this.