Use torch.nn.DataParallel for intra-node computation

[tlin-taolin]

It might be a good choice to use torch.nn.DataParallel for intra-node computation (across multi-GPUs) and intra-node gradient aggregation, and then use different communication backends for inter-node communication.

[ehoelzl]

Wouldn't it be more appropriate to use DistributedDataParallel, as referenced here ?

[tlin-taolin]

I think it should be a standard baseline for us to compare with, and we should check both of DataParallel and DistributedDataParallel for the intra-node case. But indeed I am not sure if we can directly use DistributedDataParallel in our current framework for intra-node communication (it also requires to run init_process_group).

BTW, I think the DistributedDataParallel is mainly designed/optimized for centralized training across multi-nodes but it loses the flexibility to use different communication strategies (e.g. compressed gradient), different communication topologies (e.g. ring topology for decentralized training). It is good to get the results from DistributedDataParallel (for distributed training) and compares it with our sync scheme.

[martinjaggi]

DDP overlaps gradient computation with the communication. the effect should be noticeable. how does it compare to our reference implementations?

[ehoelzl]

Closing in favor of mlbench/mlbench-benchmarks#69