Skip to content

DAWNBench Introduction

An End-to-End Deep Learning Benchmark and Competition

DAWNBench is a benchmark suite for end-to-end deep learning training and inference. Computation time and cost are critical resources in building deep models, yet many existing benchmarks focus solely on model accuracy. DAWNBench provides a reference set of common deep learning workloads for quantifying training time, training cost, inference latency, and inference cost across different optimization strategies, model architectures, software frameworks, clouds, and hardware.

DAWNBench Image Classification on CIFAR10

training multi gpu

In my test, 33 out of 50 runs reached 94% test set accuracy. Runtime for 35 epochs is roughly 38sec using Kakao Brain BrainCloud V4.XLARGE Type (V100 4GPU, 56CPU, 488GB).

trials > 94% count average median min max
metric 50                33 94.074 94.080 93.780 94.390

training single gpu

In my test, 30 out of 50 runs reached 94% test set accuracy. Runtime for 25 epochs is roughly 68sec using Kakao Brain BrainCloud V1.XLARGE Type (V100 1GPU, 56CPU, 488GB).

trials > 94% count average median min max
metric 50                30 94.057 94.045 93.700 94.330

inference latency

In my test, runs reached 94% test set accuracy. Runtime latency per sample is roughly 0.9 milliseconds using Kakao Brain BrainCloud V1.XLARGE Type (V100 1GPU, 56CPU, 488GB).

accuracy:94.20%, 0.85933us per sample

Environment Setup & Experiments

  • pre requirements
$ apt update
$ apt install -y libsm6 libxext-dev libxrender-dev libcap-dev
$ pip install torch torchvision
  • clone and init. the repository
$ git clone {THIS_REPOSITORY} && cd torchskeleton
$ pip install -r requirements.txt
  • run dawnbench image classification on CIFAR10
$ # train
$ python bin/dawnbench/cifar10.py --seed 0xC0FFEE --download > log_dawnbench_cifar10.tsv
$ python bin/dawnbench/cifar10_multigpu.py --num-gpus 4 --seed 0x00CAFE --download > log_dawnbench_cifar10_multigpu.tsv
$ # infer
$ python bin/dawnbench/cifar10_infer.py --model assets/kakaobrain_custom-resnet9_single_cifar10.pth  --download
Assets 2
You can’t perform that action at this time.