V0.6.0 Release Plan

[cp5555]

Release Manager

@cp5555

Endgame

• Code freeze: August 22nd
• Bug Bash date: August 22nd
• Release date: September 4th

Main Features

SuperBench Improvement

1. • Support running on host directly without Docker (Support sb run on host directly without Docker #358, Runner - Fix sudo issue when running without Docker #362)
2. • Support running sb command inside docker image (Dockerfile - Add support to run sb command inside docker image #356)
3. • Support ROCm 5.1.1 (Dockerfile - Add dockerfile for rocm5.1.1 #353, Dockerfile: Update rccl version and fix issue in rocm5.1.1 dockerfile #354)
4. • Support ROCm 5.1.3 (Update ROCm Dockerfile #361)
5. • Fix bugs in data diagnosis (Analyzer - Fix bugs in data diagnosis #355)
6. • Fix cmake and build issues (Fix cmake and build issues #360)
7. • Support automatic configuration yaml selection on Azure VM (CLI - Support SKU auto detect if running on Azure VM #365)
8. • Refine error message when GPU is not detected. (Deployment - Refine error message when GPU is not detected #368)
9. • Add return code for Timeout (Gracefully exit when timeout #383)
10. • Update Dockerfile for NCCL/RCCL version, tag name, and verbose output. (Update dependencies and Dockerfile #371)
11. • Support node_num=1 in mpi mode (Runner - Support node_num=1 in mpi mode #372)
12. • Update Python setup for require packages (Update Python setup for require packages #387)
13. • Enhance parameter parsing to allow spaces in value (Enhance parameter parsing to allow spaces in value #397)
14. • Support NO_COLOR for SuperBench output (Upgrade colorlog for NO_COLOR support #404)

Micro-benchmark Improvement

1. • Fix issues in ib loopback benchmark (Fix issues in ib loopback benchmark #369)
2. • Fix stability issue in ib loopback benchmark (Benchmarks: Build Pipeline - Degrade perftest submodule to fix stability issue #386)

Distributed Benchmark Improvement

1. • Pair-wise IB benchmark (Support multiple IB/GPU in ib validation #363)
2. • Bug Fix in IB benchmark (Fix issues in ib validation benchmark #370, Fix port conflict in ib loopback #375, Benchmarks: Fix Bug - Fix unexpected base conversion when the result value is negative in IB benchmark #377, Enable latency test in ib traffic validation distributed benchmark #396)
3. • Topology-aware IB benchmark (Support topo-aware IB performance validation #373, Auto generate ibstat file for IB performance validation under topo-aware … #381)

Data Diagnosis & Analysis

1. • Add failure check function in data_diagnosis.py (Analyzer - Add failure check feature in data diagnosis #378)
2. • Support Json and Jsonl in Diagnosis. (Analyzer - Add support for both jsonl and json format in data diagnosis #388)
3. • Add support to store values of metrics in data diagnosis. (Analyzer - Add support to store values of metrics in data diagnosis #392, Analyzer - Make baseline file optional in data diagnosis and fix bugs #399)
4. • Support exit code of sb result diagnosis (Analyzer - Update error handling to support exit code of sb result diagnosis #403)
5. • Format int type and unify empty value to N/A in diagnosis output files (Analyzer - Format int type and unify empty value to N/A in diagnosis output files #406)

Backlog

Inference Benchmark Improvement

1. Support VGG, LSTM, and GPT-2 small in TensorRT Inference Backend
2. Support VGG, LSTM, and GPT-2 small in ORT Inference Backend
3. Support more TensorRT parameters (Related to TensorRT parameter passing can be enhanced #366)

Data Diagnosis & Analysis

1. Support boxplot and outlier analysis

Document

1. Metric Reasoning Doc