[resnet50v1.5/tensorflow] Training performance cannot reproduce

[Haijunlv]

Related to Model/Framework(s)
resnet50v1.5/tensorflow

Describe the bug
mixed precsion+xla+batch size 256 1gpu can only get 786 img/s
mixed precsion+xla+batch size 256 4gpu get 3068 img/s.
much slower than benchmark 1270 img/s(1gpu)

To Reproduce
Steps to reproduce the behavior:
we did not change code, just use below code to start

mpiexec --allow-run-as-root --bind-to none --map-by slot -np 2 python main.py \
    --mode=training_benchmark \
	--use_xla \
	--warmup_steps 200 \
	--num_iter 500 \
	--iter_unit batch \
	--batch_size 256 \
	--data_dir=/ssd2/imagenet/tfrecord/train \
	--results_dir=${work_dirs} \
	--use_tf_amp \
	--use_static_loss_scaling \
	--loss_scale=128

we use mpiexec --allow-run-as-root --bind-to none --map-by slot -np xxx to start job because --bind-to socket will cause failed to bind memory warning and decrease speed.

and use our code start there wiil be one warning, seems like our environment donot have openib.
i donot know if this message will decrease speed.

Environment
Please provide at least:

• Container version (e.g. pytorch:19.05-py3): nvcr.io/nvidia/tensorflow:20.06-tf1-py3
• GPUs in the system: (e.g. 8x Tesla V100-SXM2-16GB): 4*Tesla V100-SXM2-32GB
• CUDA driver version (e.g. 418.67): 450.80.02, cuda11
• CPU：Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz * 16
  *CODE: NGC-20.06.5 official code

** log ***
[
log.txt
](url)

[Haijunlv]

we tried with nvcr.io/nvidia/tensorflow:20.12-tf1-py3
CUDA driver version (e.g. 418.67): 460.32.03, cuda11.2
and found the similar speed 767 imgs/s

and if we add code after https://github.com/NVIDIA/DeepLearningExamples/blob/master/TensorFlow/Classification/ConvNets/runtime/runner.py#L425

hook = tf.estimator.ProfilerHook(save_steps=5, output_dir=self.run_hparams.log_dir)
        training_hooks.append(hook)

the speed up to 952 imgs/s
but still lower than 1260 imgs/s benchmark

[Haijunlv]

we tried with nvcr.io/nvidia/tensorflow:20.12-tf1-py3 and code ngc-4
https://ngc.nvidia.com/catalog/resources/nvidia:resnet_50_v1_5_for_tensorflow/releaseNotes?version=4
using

mpiexec --allow-run-as-root --bind-to none --map-by slot -np 1 python3 main.py \
    --mode=training_benchmark \
	--use_xla \
	--warmup_steps 200 \
	--num_iter 500 \
	--iter_unit batch \
	--batch_size 256 \
	--data_dir=/ssd2/imagenet/tfrecord/train \
	--results_dir=${work_dirs} \
	--use_tf_amp \
	--use_static_loss_scaling \
	--loss_scale=128 2>&1 | tee -a ${work_dirs}/log.txt

and found speed is 1278, close to benchmarks.

so why the 20.06 code can only get 786 imgs/s, and the older code can get 1278?

[hXl3s]

Hi @Haijunlv

I've benchmarked the code on DGX1 and could not reproduce the issue on our side. The command posted gives approx 1250img/s. I used CUDA11 and 450.51.06 driver to get those results.

Adding profiler hook makes no sense in terms of improving performance measured. Profilers usually create overheads that should decrease it. Could you try the synthetic dataset setup to check if it is not a dataloading bottleneck? Removing --data_dir argument is enough to benchmark synthetic.

Best

[Haijunlv]

Hi @Haijunlv

I've benchmarked the code on DGX1 and could not reproduce the issue on our side. The command posted gives approx 1250img/s. I used CUDA11 and 450.51.06 driver to get those results.

Adding profiler hook makes no sense in terms of improving performance measured. Profilers usually create overheads that should decrease it. Could you try the synthetic dataset setup to check if it is not a dataloading bottleneck? Removing --data_dir argument is enough to benchmark synthetic.

Best

@hXl3s
we test with synthetic dataset , and found 1216 ips. then it semms dataloading is bottleneck.
first we test disk speed, found 142MB/s disk read speed, much slower than ssd.
we change to ssd disk, test with 300MB/s disk read spped, still ips is 767.
so disk read spped may not be the major bottleneck.
second we test cpu usage,
ngc-20.06-code(https://ngc.nvidia.com/catalog/resources/nvidia:resnet_50_v1_5_for_tensorflow/releaseNotes?version=20.06.5)

ngc-4-19.08-code (
https://ngc.nvidia.com/catalog/resources/nvidia:resnet_50_v1_5_for_tensorflow/releaseNotes?version=4)

cpu usage is much different between older and new code. Maybe there some config with data loader thread is changing?
in high speed disk, cpu usage 300% is enough for 1200ips,
but in lower speed disk, cpu usage have to be increased ?

--

[Haijunlv]

https://github.com/NVIDIA/DeepLearningExamples/blob/master/TensorFlow/Classification/ConvNets/runtime/runner.py#L249
this code may not be suitable for different environment,
for me ,we have only cpu num 16, and inter_op_parallelism_threads is calculated as 2.
maybe this inter_op_parallelism_threads can be setting by users
chang inter_op_parallelism_threads to 4, we got 1215 ips at ngc-20.06-code

[hXl3s]

You are right, calculation for thread count changed with 20.06, especially, there is a difference in single GPU configuration.

We cannot provide configurations for every possible hardware configuration, so reproducing results on different hardware than DGXs will always require some modifications, but I see, how this parameter could be calculated based on thread count and number of gpu available (instead of assuming that 8 gpu is available).