Skip to content
Switch branches/tags
Go to file

Latest commit


Git stats


Failed to load latest commit information.
Latest commit message
Commit time

ddl-benchmarks: Benchmarks for Distributed Deep Learning.


This repository contains a set of benchmarking scripts for evaluating the training performance of popular distributed deep learning methods in our paper, which mainly focuses on system-level optimization algorithms of synchronized stochastic gradient descent with data parallelism. Currently, it covers:

system architectures

optimization algorithms

  • Wait-free backpropagation (WFBP), which is also known as the technique of pipelining the backward computations with gradient communications and it is a default feature in current deep learning frameworks.
  • Tensor fusion, which has been integraded in Horovod with a hand-craft threshold to determine when to fuse tensors, but it is possible to dynamically determine to fuse tensors in MG-WFBP.
  • Tensor partition and priority schedule, which are proposed in ByteScheduler.
  • Gradient compression with quantization (i.e., signSGD) and sparsification (i.e., TopK-SGD). These methods are included in the code, but they are excluded from our paper as the paper focuses on the system-level optimization methods.

deep neural networks



Get the code

$git clone
$cd ddl-benchmarks 
$pip install -r requirements.txt

Configure the cluster settings

Before running the scripts, please carefully configure the configuration files in the directory of configs.

  • configs/cluster*: configure the host files for MPI
  • configs/envs.conf: configure the cluster enviroments.

Create a log folder, e.g.,

$mkdir -p logs/pcie

Run benchmarks

  • The batch mode
  • The individual mode, e.g.,
$cd horovod
$dnn=resnet50 bs=64 nworkers=64 ./


If you are using this repository for your paper, please cite our work

    author = {Shi, Shaohuai and Tang, Zhenheng and Chu, Xiaowen and Liu, Chengjian and Wang, Wei and Li, Bo},
    title = {Communication-Efficient Distributed Deep Learning: Survey, Evaluation, and Challenges},
    journal = {arXiv},
    url = {\url{}},
    year = {2020}


ddl-benchmarks: Benchmarks for Distributed Deep Learning



No releases published


No packages published