V0.3.0 Release Plan

[TobeyQin]

Release Manager

@TobeyQin

Endgame

Code freeze: 9/1/2021
Bug Bash date: 9/2/2021
Release date: 9/17/2021

Main Features

SuperBench Framework

SB Runner -- @abuccts

• MPI mode implementation
  PR: Runner - Support mpi mode #146

SB Benchmarks -- @guoshzhao

• Docker Base
  PR: Benchmarks: Code Revision - revise the DockerBenchmark base class #179 and Benchmarks: Docker Benchmarks - Setup Docker-in-Docker environment  #180

Single-node Validation

Micro-benchmarks -- @guoshzhao @yukirora

1. • Memory (Tool: Nvidia Bandwidth Test Tool) -- @yukirora ETA: 5/28/2021
     PR: Benchmarks: Add Benchmark - Add memory bandwidth benchmark for cuda. #114

     Metrics	Unit	Description
     H2D_Mem_BW_<GPU ID>	GB/s	host-to-GPU bandwidth for each GPU
     D2H_Mem_BW_<GPU ID>	GB/s	GPU-to-host bandwidth for each GPU

2. • Device P2P Bandwidth (Tool: Nvidia p2pBandwidthLatencyTest Tool) -- Delayed

     Metrics	Unit	Description
     P2P_BW_Max	GB/s	The maximum bandwidth in Bidirectional P2P=Enabled Bandwidth Matrix for all GPUs
     P2P_BW_Min	GB/s	The minimum bandwidth
     P2P_BW_Avg	GB/s	The average bandwidth

3. • IBLoopback (Tool: PerfTest – Standard RDMA Test Tool) -- @yukirora ETA: 7/30/2021
     PR: Benchmarks: Add Benchmark - Add IB Loopback performance benchmark. #112 and Benchmarks: Build Pipeline - Add perftest as a submodule and add build logic #129

     Metrics	Unit	Description
     IB_Write	MB/s	The IB write loopback throughput with different message size
     IB_Read	MB/s	The IB read loopback throughput with different message size
     IB_Send	MB/s	The IB send loopback throughput with different message size

4. • NCCL (Tool: Nvidia NCCL Test) -- @yukirora ETA: 7/30/2021
     PR: Benchmarks: Add Benchmark - Add NCCL performance benchmark. #113 and Benchmarks: Build Pipeline - Add nccl-tests as a submodule and add build logic. #128

     Metrics	Unit	Description
     NCCL_AllReduce	GB/s	The NCCL AllReduce performance with different message size
     NCCL_AllGather	GB/s	The NCCL AllGather performance with different message size
     NCCL_broadcast	GB/s	The NCCL Broadcast performance with different message size
     NCCL_reduce	GB/s	The NCCL Reduce performance with different message size
     NCCL_reduce_scatter	GB/s	The NCCL ReduceScatter performance with different message size

5. • Disk (Tool: FIO – Standard Disk Performance Tool) -- @yzygitzh ETA: 7/30/2021
     PR: Benchmarks: Build Pipeline - Add FIO benchmark tool #127 and Benchmarks: Add Benchmark - Add disk performance benchmark #132 and Benchmarks: Revise Benchmark - Add readwrite I/O pattern #161

     Metrics	Unit	Description
     Seq_Read	MB/s	Sequential read performance
     Seq_Write	MB/s	Sequential write performance
     Rand_Read	MB/s	Random read performance
     Rand_Write	MB/s	Random write performance
     Seq_R/W_Read	MB/s	Read performance in sequential read/write, fixed measurement (read:write = 4:1)
     Seq_R/W_Write	MB/s	Write performance in sequential read/write (read:write = 4:1)
     Rand_R/W_Read	MB/s	Read performance in random read/write (read:write = 4:1)
     Rand_R/W_Write	MB/s	Write performance in random read/write (read:write = 4:1)

6. • H2D/D2H SM Transmission Bandwidth (Tool: MSR-A build) -- @yzygitzh ETA: 8/6/2021
     PR: Benchmarks: Add Benchmark - Add GPU SM copy benchmark #162 and Benchmarks: Add Benchmark - Add GPU SM copy benchmark #169

     Metrics	Unit	Description
     H2D_SM_BW_<GPU ID>	GB/s	host-to-GPU bandwidth using GPU kernel for each GPU
     D2H_SM_BW_<GPU ID>	GB/s	GPU-to-host bandwidth using GPU kernel for each GPU

Support AMD

Docker Image Support -- @guoshzhao ETA: 7/16/2021

• ROCm 4.2 PyTorch 1.7 PR: Dockerfile: Add dockerfile - Add rocm 4.0 and 4.2 dockerfile with pytorch1.7.0 #164
• ROCm 4.0 PyTorch 1.7 PR: Dockerfile: Add dockerfile - Add rocm 4.0 and 4.2 dockerfile with pytorch1.7.0 #164

Micro Benchmarks

1. • Kernel Launch (Tool: MSR-A build) -- @yukirora ETA: 7/30/2021
     PR: Benchmarks: Build Pipeline - Support rocm cmake build #137 and Benchmarks: Add Benchmark - Add the source code of rocm kernel launch overhead benchmark.  #136

     Metrics	Unit	Description
     Kernel_Launch_Event_Time	Time (ms)	Dispatch latency measured in GPU time using hipEventRecord()
     Kernel_Launch_Wall_Time	Time (ms)	Dispatch latency measured in CPU time

2. • RCCL (Tool: AMD RCCL Test) -- @yukirora ETA: 7/30/2021
     PR: Benchmarks: Build Pipeline - add rccl-tests as a submodule with building logic #139 and Benchmarks: Add Benchmark - Revise and add rccl microbenchmark for rocm. #143

     Metrics	Unit	Description
     RCCL_AllReduce	GB/s	The RCCL AllReduce performance with different message size
     RCCL_AllGather	GB/s	The RCCL AllGather performance with different message size
     RCCL_broadcast	GB/s	The RCCL Broadcast performance with different message size
     RCCL_reduce	GB/s	The RCCL Reduce performance with different message size
     RCCL_reduce_scatter	GB/s	The RCCL ReduceScatter performance with different message size

3. • GEMM FLOPS (Tool: AMD rocblas-bench Tool) -- @yukirora ETA: 8/27/2021
     PR: Benchmarks: Build Pipeline - Add rocBLAS building logic in third_party #144 and Benchmarks: Code Revision - Extract base class for gemm flops microbenchmark #165

     Metrics	Unit	Description
     FP64	GFLOPS	FP64 FLOPS without MatrixCore
     FP32	GFLOPS	FP32 FLOPS without MatrixCore
     FP16	GFLOPS	FP16 FLOPS without MatrixCore
     FP32(MC)	GFLOPS	TF32 FLOPS with MatrixCore
     FP16(MC)	GFLOPS	FP16 FLOPS with MatrixCore
     BF16(MC)	GFLOPS	BF16 FLOPS with MatrixCore
     INT8(MC)	GOPS	INT8 FLOPS with MatrixCore
     INT4(MC)	GOPS	INT4 FLOPS with MatrixCore

4. • Memory (Tool: HIP Bandwidth Test Tool) -- @yukirora ETA: 8/27/2021
     PR: Benchmarks: Code Revision - Extract base class for memory bandwidth microbenchmark #159 and Benchmarks: Add Benchmark - Add memory bus bandwidth performance microbenchmark for amd  #153

     Metrics	Unit	Description
     H2D_Mem_BW_<GPU ID>	GB/s	host-to-GPU bandwidth for each GPU
     D2H_Mem_BW_<GPU ID>	GB/s	GPU-to-host bandwidth for each GPU

E2E Benchmarks -- @guoshzhao ETA: 7/16/2021

1. • CNN models -- User PyTorch TORCHVISION.MODELS sub-package
     • ResNet: ResNet-50, ResNet-101, ResNet-152
     • DenseNet: DenseNet-169, DenseNet-201 ​
     • VGG: VGG-11, VGG-13, VGG-16, VGG-19​
2. • BERT -- Use huggingface Transformers
     • BERT
     • BERT LARGE
3. • LSTM -- Use PyTorch TORCH.NN sub-package
4. • GPT-2 -- Use huggingface Transformers

Result Summary -- @cp5555

• Generate a report to summarize the results -- @guoshzhao ETA: 7/30/2021
  PR: Benchmarks: Add Feature - Add reduce function support for output summary. #147, Benchmarks: Add Feature - Set reduce type for current benchmarks' metrics. #149, and Runner: Add Feature - Generate summarized output files. #157
• Support basic analysis feature (boxplot figure, outlier detection, etc.)

Bug Fix

• VGG models failed on A100 GPU with batch_size=128 VGG models failed on A100 GPU with batch_size=128 #115
  PR: Benchmarks: Fix Bug - Fix bug of VGG models failed on A100 GPU with batch_size=128 #134

Other Improvement

1. Contribution related -- @lynex

   • Contribute rule (Docs - Add result contributing rules #131)
   • system information collection (Tools: Add Feature - Add script to generate system config info. #160)

2. Document -- @TobeyQin

   • Add release process doc (Docs - Add release process doc #130)
   • Add design documents (Docs - Add design document #125)
   • Add developer guide doc for coding style (Docs - Add config and docs for development experience #155)
   • Add contribution rules (Docs - Add result contributing rules #131)
   • Add docker image list (Docs - Add docs for Docker container and image #154)
   • Add initial validation results
   • Add metric reasoning doc -- @cp5555 @guoshzhao

3. Process monitor

   • Add Heart beat to monitor process health
   • Auto kill all processes on all nodes

4. Coding style -- @abuccts

   • Add vscode online

---

Backlogs

Multi-Node Benchmarks

• Mellanox ClusterKit
• GPCNeT

UI Design