Performance doesn't improve (scalability issue) with # GPUs with running train_imagenet.py

[rohith14]

While training AlexNet CNN with ImageNet data, i don't see performance improvement (in-fact i see slight performance degradation) with increasing number of GPUs

python train_imagenet.py --data-train /local/ImageNet/MXNet_data/MXNet_data.rec --data-val /local/ImageNet/MXNet_data/MXNet_data_test.rec --gpus 0,1,2,3 --network alexnet --batch-size 256 --num-epochs 1 --kv-store device

Per epoch (and batch-size/GPU : 64),
With 1 GPU, Time-cost : 910 sec
With 2 GPU, Time-cost : 924 sec
With 4 GPU, Time-cost : 964 sec

I have 4 Titan Xps

However, with synthetic data (as shown in the demo https://github.com/apache/incubator-mxnet/blob/master/example/image-classification/README.md) i see good scalability.

[ptrendx]

You are CPU limited - Alexnet is a very small network and Titans are fast. How many cores do you have on your CPU? If more than 4 then try to set --data-nthreads N where N is number of threads to do IO. By default MXNet uses only 4 cores to do IO. You may also limit augmentations you perform (by default train_imagenet script performs a lot of augmentations).
I am in the process of improving image IO (#7152), that should also help.

[rohith14]

Thank you, ptrendx. That helps, but with 4 GPUs there isn't any performance gain

% i limited the augmentations by setting 'data.set_data_aug_level(parser, 0)'
%set data_threads to 8 (I have 8 physical cores)
python train_imagenet.py --data-train /local/ImageNet/MXNet_data/MXNet_data.rec --data-val /local/ImageNet/MXNet_data/MXNet_data_test.rec --gpus 0,1,2,3 --network alexnet --data-nthreads 8 --batch-size 1024 --num-epochs 1 --kv-store device

With 1 GPU, Time-cost : 680 sec
With 2 GPU, Time-cost : 373 sec
With 4 GPU, Time-cost : 378 sec

[ptrendx]

As I said, alexnet is a very small network, so with 8 physical cores you probably will not get a much better scaling result with those GPUs. Try a bigger network (vgg, resnet 50 or something similar) where there is not as big stress on IO and you should get a very good scaling result.

Still, IO speed is a big problem going forward even for larger networks and that is why we are working on ways to improve it (besides my PR see for example #7092)