V0.6.0 Test Plan

[yukirora]

V0.6.0 Test Plan

Test Cases

single-node test

Machine Type	#Node * #GPU * GPU Type	PyTorch Version	Accelerated Computing Toolkit	Status
ND A100 v4	1 * 8 * A100 40GB SXM	PyTorch 1.8	CUDA 11.1	Done
NDm A100 v4	1 * 8 * A100 80GB SXM	PyTorch 1.8	CUDA 11.1	Done
Hayabusa	1* 16 * MI200	PyTorch 1.9	ROCm 5.1	Done

single-node Micro-benchmark Test

1. ib-loopback

• Fix issues in ib loopback benchmark (Fix issues in ib loopback benchmark #369)
• Fix stability issue in ib loopback benchmark (Benchmarks: Build Pipeline - Degrade perftest submodule to fix stability issue #386)
• fix port conflict in ib loopback (Fix port conflict in ib loopback #375)

2. Rccl-test/nccl-test

• Update Dockerfile for NCCL/RCCL version, tag name, and verbose output (Update dependencies and Dockerfile #371)
• Support node_num=1 in mpi mode(Runner - Support node_num=1 in mpi mode #372)

SuperBench Improvement

• Support running on host directly without Docker(Dockerfile - Add support to run sb command inside docker image #356, Support sb run on host directly without Docker #358, Runner - Fix sudo issue when running without Docker #362)
• Support automatic configuration yaml selection on Azure VM
• Add return code for Timeout(Gracefully exit when timeout #383,Runner - Fix minimum timeout #385)
• Support ROCm 5.1.1 (Dockerfile - Add dockerfile for rocm5.1.1 #353, Dockerfile: Update rccl version and fix issue in rocm5.1.1 dockerfile #354), Support ROCm 5.1.3 (Update ROCm Dockerfile #361)

Tools

1. data diagnosis

• Fix bugs in data diagnosis (Analyzer - Fix bugs in data diagnosis #355)
• Add failure check function in data_diagnosis.py (Analyzer - Add failure check feature in data diagnosis #378)
• Support Json and Jsonl in Diagnosis. (Analyzer - Add support for both jsonl and json format in data diagnosis #388)
• Add support to store values of metrics in data diagnosis. (Analyzer - Add support to store values of metrics in data diagnosis #392)

New in bug bash

• Make baseline file optional in data diagnosis and fix bugs (Analyzer - Make baseline file optional in data diagnosis and fix bugs #399)
• Update error handling to support exit code of sb result diagnosis (Analyzer - Update error handling to support exit code of sb result diagnosis #403)
• Format int type and unify empty value to N/A in diagnosis output files (Analyzer - Format int type and unify empty value to N/A in diagnosis output files #406)
• Upgrade colorlog for NO_COLOR support (Upgrade colorlog for NO_COLOR support #404)
• Enhance timeout cleanup to avoid possible hanging (Enhance timeout cleanup to avoid possible hanging #405)

multiple-node test

Test Table

Machine Type	#Node * #GPU * GPU Type	PyTorch Version	Accelerated Computing Toolkit	Status
ND A100 v4	32 * 8 * A100 40GB SXM	PyTorch 1.8	CUDA 11.1	Done

distributed Micro-benchmark test

1. ib-traffic

• Support multiple IB/GPU Pair-wise IB benchmark (Support multiple IB/GPU in ib validation #363)
• Bug Fix in IB benchmark in all-pair mode(Fix issues in ib validation benchmark #370, Benchmarks: Fix Bug - Fix unexpected base conversion when the result value is negative in IB benchmark #377)
• Topology-aware IB benchmark (Support topo-aware IB performance validation #373, Auto generate ibstat file for IB performance validation under topo-aware … #381)

New in bug bash

• Auto generate ibstat file by pssh (Auto generate ibstat file by pssh #402)
• Enable latency test in ib traffic validation distributed benchmark(Enable latency test in ib traffic validation distributed benchmark #396)