throughput drop when train resnet50

[wchen61]

Hi, @rwightman.
When I run resnet50 on my A100, I notice throughput will drop once in a while.

I use the following command to run resnet50 on A100 with NGC22.04:

python train.py ./imagenet -b 512 --sched cosine --epochs 1 --lr 0.05 --amp --channels-last --model resnet50 -j10 --pin-mem --log-interval 1

And if I run with benchmark.py, I can get good result, throughput always 2600 img/s.

[rwightman]

@wchen61 It's normal, there are hiccups in the instantaneous throughout, you should be trying to improve the avg which is only 1923. The gap between last step and avg is fairly high so there is likely a bottleneck. Not a timm bug. Make sure you've got pillow-simd installed and no normal pillow in the env. Use a fast ssd for dataset or use TFDS.

[wchen61]

@wchen61 It's normal, there are hiccups in the instantaneous throughout, you should be trying to improve the avg which is only 1923. The gap between last step and avg is fairly high so there is likely a bottleneck. Not a timm bug. Make sure you've got pillow-simd installed and no normal pillow in the env. Use a fast ssd for dataset or use TFDS.

After I use the pillow-simd, I can get throughout of 2500 img/s. Thank you so much for saving my day! @rwightman