Skip to content

DAWNBench Introduction

An End-to-End Deep Learning Benchmark and Competition

DAWNBench is a benchmark suite for end-to-end deep learning training and inference. Computation time and cost are critical resources in building deep models, yet many existing benchmarks focus solely on model accuracy. DAWNBench provides a reference set of common deep learning workloads for quantifying training time, training cost, inference latency, and inference cost across different optimization strategies, model architectures, software frameworks, clouds, and hardware.

DAWNBench Image Classification on CIFAR10

training multi gpu

In my test, 33 out of 50 runs reached 94% test set accuracy. Runtime for 35 epochs is roughly 29sec using Kakao Brain BrainCloud V4.XLARGE Type (V100 4GPU, 56CPU, 488GB).

trials > 94% count average median min max
metric 50                33 94.074 94.080 93.780 94.390

training single gpu

In my test, 30 out of 50 runs reached 94% test set accuracy. Runtime for 25 epochs is roughly 57sec using Kakao Brain BrainCloud V1.XLARGE Type (V100 1GPU, 56CPU, 488GB).

trials > 94% count average median min max
metric 50                30 94.057 94.045 93.700 94.330

inference latency

In my test, runs reached 94% test set accuracy. Runtime latency per sample is roughly 0.9 milliseconds using Kakao Brain BrainCloud V1.XLARGE Type (V100 1GPU, 56CPU, 488GB).

accuracy:94.20%, 0.85933us per sample
Assets 2
You can’t perform that action at this time.