[Question] Inference time speed up or not?

[tuanmanh1410]

Thank for sharing this project and paper
I'm using GPipe Pytorch to testing inference time the same test dataset and compare with running on single GPU as baseline.
1/ The inference time running with GPiPe is seem slower than single GPU. Therefore , GPipe is suitable for training large model ? and not effective for speed up inference time ? Please correct me if I'm wrong.
2/ I'm curious that Does GPipe library support computes the communication latency among GPUs when intermediate data is transmitted between 2 GPUs in a row?

Thank you

[sublee]

The purpose of GPipe is:

• Be able to train or evaluate large model which cannot be placed on a single GPU.
• Reduce GPU idle time during model parallelism by pipelining. But there is still idle time. We call it "bubble".

If your model isn't large enough, you don't need GPipe.

For more details, see:

• https://torchgpipe.readthedocs.io/en/stable/gpipe.html#pipeline-parallelism
• https://pytorch.org/docs/stable/pipeline.html

Also, GPipe itself does not provide latency measurement. We'd used NVIDIA Nsight Systems to optimize its communication cost.