Skip to content

Latest commit

 

History

History
309 lines (278 loc) · 40.3 KB

README.md

File metadata and controls

309 lines (278 loc) · 40.3 KB
Raw

MPT Training Benchmarks

Benchmark measurements for MPT models trained on MosaicML platform, including throughput, MFU, and HFU. Each model is based on optimized configurations of various sizes in the yamls folder, ranging from a 125m to 70B parameter models.

To reproduce table results, first run:

./sweep.sh

Then, after the runs are completed:

python collect_results.py --save-path results

will use our Python API to collect and calculate the benchmark results, and then save as both a CSV file results.csv, and a markdown table results.md.

python submit_benchmarks.py --cluster [your_mosaicml_cluster] ARGS --RUN

can be used to sweep a larger set of configurations. For example usage of submit_benchmarks.py see sweep.sh which lists all benchmarks in the tables.

Note The collect_results.py will by default find all runs with tput in the run name. To customize this project tag, use --project in both the submission and collection scripts.

MFU and HFU

Model FLOPs Utilization (MFU) and Hardware FLOPS Utilization (HFU) are estimates, based on the measured throughput and the known FLOPs of the computation, of what percentage of the hardware's FLOPs are being used during training.

MFU calculates the utilization from the floating point operations required for a single forward/backwards pass of the model, and does not account for the additional compute required for other implementation details such as activation checkpointing. Thus, MFU is independent of implementation and hardware.

HFU attempts to capture the actual floating point operations incurred during the forward/backwards pass on the hardware. While it is a more accurate measurement of hardware utilization, it is less general and is difficult to compare across various hardware and implementation details.

For more information, see Korthikanti et al, 2022. All FLOP calculations exclude the operations required for normalization, activation, and residuals.

MFU

Per token, each parameter is used for a MAC (2 FLOPS) per network operation. Neural Network training has 3 network operations: forward pass, backward pass, and computation of parameter gradient.

The attention mechanism forward pass FLOPS are: attn_flops_per_seq = n_layers * 2 * 2 * (d_model * (seq_len**2))

flops_per_token = 2 * n_params
flops_per_seq = flops_per_token * seq_len
mfu* = 3 * flops_per_seq * seq_per_sec / (gpu_num * GPU_AVAILABLE_FLOPS)

attn_flops_per_seq = n_layers * 2 * 2 * (d_model * (seq_len**2))
mfu = (3 * flops_per_seq + 3 * attn_flops_per_seq) * seq_per_sec / (gpu_num * GPU_AVAILABLE_FLOPS)

HFU

The HFU numbers shown below account for the fact that the networks use checkpointing and recomputes activations. This effectively requires an extra forward pass through the network.

hfu* = 4 * flops_per_seq * seq_per_sec / (gpu_num * GPU_AVAILABLE_FLOPS)
hfu = (4 * flops_per_seq + 4 * attn_flops_per_seq) * seq_per_sec / (gpu_num * GPU_AVAILABLE_FLOPS)

Note that these are approximations. Actual HFU would be higher since it includes the floating point operations for normalization, activation, and residual layers, as well as all recomputation. For example, our models use Flash Attention, which requires including an extra recompute factor for its recomputation in the forward pass. Therefore, the attention multiplier would be 5 instead of 4.

Results

Below we include several configurations across different hardware platforms, sequence lengths and batch sizes. It is easy to benchmark configurations for your own use case. For example, using the Mosaic platform, to test MPT {13B, 30B} using fp16 with a batch size of 2M tokens and seq len {2k, 4k, 8k, 16k} run:

python submit_benchmarks.py -m 13b.yaml 30b.yaml -t fp16 -b 21 21 -s 11 14 --RUN

This will run 8 configs for 12 steps to get throughput numbers. python collect_results.py can then be used to parse all output training logs and create the tables below.

Our microbatching engine enables microbatch sizes that do not divide global batch size while being mathematically faithful to the global batch size. For example, a total batch size of 48, and a micro batch of 11, means we will accumulate gradients across microbatches of 11, 11, 11, 11, 4.

H100 80GB BF16 (Large Scale, >= 128 GPUs)

Model SeqLen (T) # GPUs GPU MFU HFU Model TFLOP MicroBatchSize GradAccum GlobalBatchSize Throughput (S/s) Throughput (T/s) Throughput (T/s/GPU) GlobalBatchSize (T) Precision MP Mode Sharding Strategy Activation Checkpointing Activation CPUOffload NumParams
70b 2048 512 h100_80gb 41.25 55.0 408 8 1 4096 251 515636 1007 8388608 amp_bf16 DEFAULT FULL_SHARD True False 64862437376
70b 2048 256 h100_80gb 42.42 56.56 419 8 1 2048 129 265149 1035 4194304 amp_bf16 DEFAULT FULL_SHARD True False 64862437376
70b 2048 128 h100_80gb 43.36 57.81 428 8 1 1024 66 135490 1058 2097152 amp_bf16 DEFAULT FULL_SHARD True False 64862437376
30b 2048 512 h100_80gb 40.27 53.69 398 8 1 4096 528 1083366 2115 8388608 amp_bf16 DEFAULT FULL_SHARD True False 29975214080
30b 2048 256 h100_80gb 40.89 54.52 404 8 1 2048 268 550022 2148 4194304 amp_bf16 DEFAULT FULL_SHARD True False 29975214080
30b 2048 128 h100_80gb 41.85 55.8 414 8 1 1024 137 281491 2199 2097152 amp_bf16 DEFAULT FULL_SHARD True False 29975214080
13b 2048 512 h100_80gb 41.12 54.83 406 16 1 8192 1238 2535811 4952 16777216 amp_bf16 DEFAULT FULL_SHARD True False 12853954560
13b 2048 256 h100_80gb 41.42 55.23 409 16 1 4096 623 1277214 4989 8388608 amp_bf16 DEFAULT FULL_SHARD True False 12853954560
13b 2048 128 h100_80gb 42.18 56.24 417 16 1 2048 317 650264 5080 4194304 amp_bf16 DEFAULT FULL_SHARD True False 12853954560
7b 2048 512 h100_80gb 42.2 42.2 417 6 1 3072 2417 4951479 9670 6291456 amp_bf16 DEFAULT FULL_SHARD False False 6658859008
7b 2048 256 h100_80gb 44.15 44.15 436 6 1 1536 1264 2590548 10119 3145728 amp_bf16 DEFAULT FULL_SHARD False False 6658859008
7b 2048 128 h100_80gb 45.71 45.71 452 6 1 768 654 1340830 10475 1572864 amp_bf16 DEFAULT FULL_SHARD False False 6658859008
3b 2048 512 h100_80gb 39.24 39.24 388 8 1 4096 5416 11092218 21664 8388608 amp_bf16 DEFAULT SHARD_GRAD_OP False False 2651837440
3b 2048 256 h100_80gb 41.25 41.25 408 8 1 2048 2846 5829686 22772 4194304 amp_bf16 DEFAULT SHARD_GRAD_OP False False 2651837440
3b 2048 128 h100_80gb 42.43 42.43 419 8 1 1024 1463 2998098 23422 2097152 amp_bf16 DEFAULT SHARD_GRAD_OP False False 2651837440
1b 2048 512 h100_80gb 36.65 36.65 362 12 1 6144 9959 20396905 39837 12582912 amp_bf16 DEFAULT SHARD_GRAD_OP False False 1315950592
1b 2048 256 h100_80gb 39.15 39.15 387 12 1 3072 5319 10894207 42555 6291456 amp_bf16 DEFAULT SHARD_GRAD_OP False False 1315950592
1b 2048 128 h100_80gb 40.6 40.6 401 12 1 1536 2757 5647854 44123 3145728 amp_bf16 DEFAULT SHARD_GRAD_OP False False 1315950592

H100 80GB BF16

Model SeqLen (T) # GPUs GPU MFU HFU Model TFLOP MicroBatchSize GradAccum GlobalBatchSize Throughput (S/s) Throughput (T/s) Throughput (T/s/GPU) GlobalBatchSize (T) Precision MP Mode Sharding Strategy Activation Checkpointing Activation CPUOffload NumParams
70b 2048 64 h100_80gb 42.57 56.76 421 8 4 2048 32 66523 1039 4194304 amp_bf16 DEFAULT FULL_SHARD True False 64862437376
70b 2048 32 h100_80gb 36.15 48.2 357 2 16 1024 13 28242 882 2097152 amp_bf16 DEFAULT FULL_SHARD True False 64862437376
30b 8192 8 h100_80gb 29.92 39.9 296 1 21 168 1 11072 1384 1376256 amp_bf16 DEFAULT FULL_SHARD True False 30019254272
30b 4096 8 h100_80gb 35.86 47.81 354 1 21 168 3 14419 1802 688128 amp_bf16 DEFAULT FULL_SHARD True False 29989894144
30b 2048 32 h100_80gb 43.92 58.57 434 14 3 1344 36 73860 2308 2752512 amp_bf16 DEFAULT FULL_SHARD True False 29975214080
30b 2048 16 h100_80gb 43.07 57.42 426 10 3 480 17 36209 2263 983040 amp_bf16 DEFAULT FULL_SHARD True False 29975214080
30b 2048 8 h100_80gb 38.11 50.82 377 3 21 504 7 16022 2002 1032192 amp_bf16 DEFAULT FULL_SHARD True False 29975214080
30b 1024 8 h100_80gb 38.76 51.68 383 6 21 1008 16 16672 2084 1032192 amp_bf16 DEFAULT FULL_SHARD True False 29967874048
13b 32768 8 h100_80gb 31.68 42.24 313 1 3 24 0 15812 1976 786432 amp_bf16 DEFAULT FULL_SHARD True False 13011240960
13b 16384 8 h100_80gb 35.55 47.4 351 3 3 72 1 23881 2985 1179648 amp_bf16 DEFAULT FULL_SHARD True False 12927354880
13b 4096 8 h100_80gb 41.6 55.47 411 10 3 240 9 37740 4717 983040 amp_bf16 DEFAULT FULL_SHARD True False 12864440320
13b 2048 64 h100_80gb 39.86 39.86 394 2 1 128 150 307209 4800 262144 amp_bf16 DEFAULT FULL_SHARD False False 12853954560
13b 2048 32 h100_80gb 39.95 39.95 395 2 1 64 75 153960 4811 131072 amp_bf16 DEFAULT FULL_SHARD False False 12853954560
13b 2048 16 h100_80gb 39.58 39.58 391 2 1 32 37 76280 4767 65536 amp_bf16 DEFAULT FULL_SHARD False False 12853954560
13b 2048 8 h100_80gb 39.79 39.79 393 2 1 16 18 38336 4792 32768 amp_bf16 DEFAULT FULL_SHARD False False 12853954560
13b 1024 8 h100_80gb 44.27 59.03 438 40 3 960 42 44019 5502 983040 amp_bf16 DEFAULT FULL_SHARD True False 12848711680
7b 65536 8 h100_80gb 28.59 38.13 282 1 2 16 0 15654 1956 1048576 amp_bf16 DEFAULT FULL_SHARD True False 6918905856
7b 32768 8 h100_80gb 30.94 41.25 306 2 2 32 0 26550 3318 1048576 amp_bf16 DEFAULT FULL_SHARD True False 6784688128
7b 8192 8 h100_80gb 37.14 49.52 367 8 2 128 6 55481 6935 1048576 amp_bf16 DEFAULT FULL_SHARD True False 6684024832
7b 4096 8 h100_80gb 40.42 53.9 399 16 2 256 16 68893 8611 1048576 amp_bf16 DEFAULT FULL_SHARD True False 6667247616
7b 2048 8 h100_80gb 46.44 46.44 459 6 1 48 41 85144 10643 98304 amp_bf16 DEFAULT FULL_SHARD False False 6658859008
7b 1024 8 h100_80gb 42.83 57.11 423 64 2 1024 79 81628 10203 1048576 amp_bf16 DEFAULT FULL_SHARD True False 6654664704
3b 65536 8 h100_80gb 26.81 35.74 265 1 2 16 0 26099 3262 1048576 amp_bf16 DEFAULT FULL_SHARD True False 2814366720
3b 32768 8 h100_80gb 28.84 38.46 285 3 6 144 1 46984 5873 4718592 amp_bf16 DEFAULT FULL_SHARD True False 2730480640
3b 16384 8 h100_80gb 36.34 36.34 359 1 6 48 5 89223 11152 786432 amp_bf16 DEFAULT FULL_SHARD False False 2688537600
3b 8192 8 h100_80gb 40.31 40.31 398 3 6 144 16 132626 16578 1179648 amp_bf16 DEFAULT FULL_SHARD False False 2667566080
3b 4096 8 h100_80gb 42.31 42.31 418 5 6 240 40 167712 20964 983040 amp_bf16 DEFAULT FULL_SHARD False False 2657080320
3b 2048 64 h100_80gb 40.8 40.8 403 6 3 1152 703 1441663 22525 2359296 amp_bf16 DEFAULT FULL_SHARD False False 2651837440
3b 2048 32 h100_80gb 41.7 41.7 412 6 3 576 359 736701 23021 1179648 amp_bf16 DEFAULT FULL_SHARD False False 2651837440
3b 2048 16 h100_80gb 43.73 43.73 432 10 3 480 188 386285 24142 983040 amp_bf16 DEFAULT FULL_SHARD False False 2651837440
3b 1024 8 h100_80gb 46.2 46.2 457 20 6 960 211 216369 27046 983040 amp_bf16 DEFAULT FULL_SHARD False False 2649216000
3b 512 8 h100_80gb 46.32 46.32 458 40 6 1920 436 223721 27965 983040 amp_bf16 DEFAULT FULL_SHARD False False 2647905280
1b 65536 8 h100_80gb 26.34 35.12 260 1 2 16 0 44050 5506 1048576 amp_bf16 DEFAULT FULL_SHARD True False 1445974016
1b 32768 8 h100_80gb 33.54 33.54 331 1 4 32 2 96203 12025 1048576 amp_bf16 DEFAULT FULL_SHARD False False 1378865152
1b 16384 8 h100_80gb 35.22 35.22 348 2 4 64 9 157194 19649 1048576 amp_bf16 DEFAULT FULL_SHARD False False 1345310720
1b 8192 8 h100_80gb 37.73 37.73 373 3 4 96 28 233256 29157 786432 amp_bf16 DEFAULT FULL_SHARD False False 1328533504
1b 4096 8 h100_80gb 40.26 40.26 398 7 4 224 75 308282 38535 917504 amp_bf16 DEFAULT FULL_SHARD False False 1320144896
1b 2048 64 h100_80gb 40.85 40.85 404 20 1 1280 1387 2841754 44402 2621440 amp_bf16 DEFAULT FULL_SHARD False False 1315950592
1b 2048 32 h100_80gb 41.52 41.52 410 20 1 640 705 1444183 45130 1310720 amp_bf16 DEFAULT FULL_SHARD False False 1315950592
1b 2048 16 h100_80gb 42.36 42.36 419 20 1 320 359 736596 46037 655360 amp_bf16 DEFAULT FULL_SHARD False False 1315950592
1b 2048 8 h100_80gb 41.82 41.82 413 14 1 112 177 363645 45455 229376 amp_bf16 DEFAULT FULL_SHARD False False 1315950592
1b 1024 8 h100_80gb 41.95 41.95 415 18 4 576 382 391287 48910 589824 amp_bf16 DEFAULT FULL_SHARD False False 1313853440
1b 512 8 h100_80gb 43.21 43.21 427 56 4 1792 816 418201 52275 917504 amp_bf16 DEFAULT FULL_SHARD False False 1312804864
760m 32768 8 h100_80gb 31.84 31.84 315 1 2 16 3 130333 16291 524288 amp_bf16 DEFAULT FULL_SHARD False False 807656448
760m 16384 8 h100_80gb 33.57 33.57 332 3 2 48 13 222521 27815 786432 amp_bf16 DEFAULT FULL_SHARD False False 782490624
760m 8192 8 h100_80gb 34.84 34.84 344 6 2 96 40 334602 41825 786432 amp_bf16 DEFAULT FULL_SHARD False False 769907712
760m 4096 8 h100_80gb 35.83 35.83 354 12 2 192 108 443674 55459 786432 amp_bf16 DEFAULT FULL_SHARD False False 763616256
760m 2048 32 h100_80gb 37.57 37.57 371 24 1 768 1062 2175091 67971 1572864 amp_bf16 DEFAULT FULL_SHARD False False 760470528
760m 2048 16 h100_80gb 37.89 37.89 374 24 1 384 535 1096819 68551 786432 amp_bf16 DEFAULT FULL_SHARD False False 760470528
760m 2048 8 h100_80gb 34.9 34.9 345 24 2 384 246 505177 63147 786432 amp_bf16 DEFAULT FULL_SHARD False False 760470528
760m 1024 8 h100_80gb 39.76 39.76 393 48 2 768 613 628648 78581 786432 amp_bf16 DEFAULT FULL_SHARD False False 758897664
760m 512 8 h100_80gb 40.42 40.42 399 96 2 1536 1308 669998 83749 786432 amp_bf16 DEFAULT FULL_SHARD False False 758111232

H100 80GB FP8

Model SeqLen (T) # GPUs GPU MFU HFU Model TFLOP MicroBatchSize GradAccum GlobalBatchSize Throughput (S/s) Throughput (T/s) Throughput (T/s/GPU) GlobalBatchSize (T) Precision MP Mode Sharding Strategy Activation Checkpointing Activation CPUOffload NumParams
3b 32768 8 h100_80gb 14.38 19.18 284 3 6 144 1 46853 5856 4718592 amp_fp8 DEFAULT FULL_SHARD True False 2730480640
3b 8192 8 h100_80gb 23.28 23.28 460 3 6 144 18 153174 19146 1179648 amp_fp8 DEFAULT FULL_SHARD False False 2667566080
3b 2048 8 h100_80gb 27.7 27.7 548 10 6 480 119 244692 30586 983040 amp_fp8 DEFAULT FULL_SHARD False False 2651837440
3b 512 8 h100_80gb 30.25 30.25 598 40 6 1920 570 292217 36527 983040 amp_fp8 DEFAULT FULL_SHARD False False 2647905280
1b 32768 8 h100_80gb 17.55 17.55 347 1 4 32 3 100643 12580 1048576 amp_fp8 DEFAULT FULL_SHARD False False 1378865152
1b 8192 8 h100_80gb 20.71 20.71 409 2 4 64 31 256087 32010 524288 amp_fp8 DEFAULT FULL_SHARD False False 1328533504
1b 512 8 h100_80gb 29.06 29.06 575 56 4 1792 1098 562523 70315 917504 amp_fp8 DEFAULT FULL_SHARD False False 1312804864

A100 80GB with 1600 Gbps node-node interconnect (RoCE)

Model SeqLen (T) # GPUs GPU MFU HFU Model TFLOP MicroBatchSize GradAccum GlobalBatchSize Throughput (S/s) Throughput (T/s) Throughput (T/s/GPU) GlobalBatchSize (T) Precision MP Mode Sharding Strategy Activation Checkpointing Activation CPUOffload NumParams
70b 2048 64 a100_80gb 53.33 71.1 166 8 4 2048 12 26274 410 4194304 bf16 PURE FULL_SHARD True False 64862437376
70b 2048 32 a100_80gb 48.56 64.75 151 2 16 1024 5 11962 373 2097152 bf16 PURE FULL_SHARD True False 64862437376
30b 8192 8 a100_80gb 39.38 52.5 122 1 21 168 0 4594 574 1376256 amp_bf16 DEFAULT FULL_SHARD True False 30019254272
30b 4096 8 a100_80gb 51.37 68.49 160 1 21 168 1 6513 814 688128 amp_bf16 DEFAULT FULL_SHARD True False 29989894144
30b 2048 8 a100_80gb 55.3 73.74 172 3 21 504 3 7330 916 1032192 amp_bf16 DEFAULT FULL_SHARD True False 29975214080
30b 1024 8 a100_80gb 55.82 74.43 174 6 21 1008 7 7571 946 1032192 amp_bf16 DEFAULT FULL_SHARD True False 29967874048
30b 512 8 a100_80gb 56.4 75.2 175 12 21 2016 15 7739 967 1032192 amp_bf16 DEFAULT FULL_SHARD True False 29964204032
13b 32768 8 a100_80gb 51.69 68.92 161 1 3 24 0 8134 1016 786432 amp_bf16 DEFAULT FULL_SHARD True False 13011240960
13b 16384 8 a100_80gb 54.07 72.1 168 3 3 72 0 11454 1431 1179648 amp_bf16 DEFAULT FULL_SHARD True False 12927354880
13b 8192 8 a100_80gb 56.07 74.76 174 5 3 120 1 14362 1795 983040 amp_bf16 DEFAULT FULL_SHARD True False 12885411840
13b 4096 8 a100_80gb 57.62 76.82 179 10 3 240 4 16482 2060 983040 amp_bf16 DEFAULT FULL_SHARD True False 12864440320
13b 2048 8 a100_80gb 59.57 59.57 185 2 3 48 8 18097 2262 98304 amp_bf16 DEFAULT FULL_SHARD False False 12853954560
13b 1024 8 a100_80gb 59.48 79.3 185 40 3 960 18 18647 2330 983040 amp_bf16 DEFAULT FULL_SHARD True False 12848711680
7b 65536 8 a100_80gb 46.97 62.63 146 1 2 16 0 8108 1013 1048576 amp_bf16 DEFAULT FULL_SHARD True False 6918905856
7b 32768 8 a100_80gb 49.46 65.94 154 2 2 32 0 13382 1672 1048576 amp_bf16 DEFAULT FULL_SHARD True False 6784688128
7b 16384 8 a100_80gb 51.96 69.28 162 4 2 64 1 19629 2453 1048576 amp_bf16 DEFAULT FULL_SHARD True False 6717579264
7b 8192 8 a100_80gb 54.47 72.62 169 8 2 128 3 25655 3206 1048576 amp_bf16 DEFAULT FULL_SHARD True False 6684024832
7b 4096 8 a100_80gb 54.84 73.12 171 16 2 256 7 29472 3684 1048576 amp_bf16 DEFAULT FULL_SHARD True False 6667247616
7b 2048 8 a100_80gb 64.23 64.23 200 6 2 96 18 37130 4641 196608 amp_bf16 DEFAULT FULL_SHARD False False 6658859008
7b 1024 8 a100_80gb 58.01 77.35 180 64 2 1024 34 34857 4357 1048576 amp_bf16 DEFAULT FULL_SHARD True False 6654664704
3b 65536 8 a100_80gb 46.05 61.41 143 1 2 16 0 14137 1767 1048576 amp_bf16 DEFAULT FULL_SHARD True False 2814366720
3b 32768 8 a100_80gb 47.18 62.91 147 3 6 144 0 24235 3029 4718592 amp_bf16 DEFAULT FULL_SHARD True False 2730480640
3b 16384 8 a100_80gb 57.13 57.13 178 1 6 48 2 44233 5529 786432 amp_bf16 DEFAULT FULL_SHARD False False 2688537600
3b 8192 8 a100_80gb 59.34 59.34 185 3 6 144 7 61567 7695 1179648 amp_bf16 DEFAULT FULL_SHARD False False 2667566080
3b 4096 8 a100_80gb 60.53 60.53 188 5 6 240 18 75658 9457 983040 amp_bf16 DEFAULT FULL_SHARD False False 2657080320
3b 2048 8 a100_80gb 62.11 62.11 193 10 2 160 42 86491 10811 327680 amp_bf16 DEFAULT FULL_SHARD False False 2651837440
3b 1024 8 a100_80gb 62.73 62.73 195 20 6 960 90 92643 11580 983040 amp_bf16 DEFAULT FULL_SHARD False False 2649216000
3b 512 8 a100_80gb 63.71 63.71 198 40 6 1920 189 97019 12127 983040 amp_bf16 DEFAULT FULL_SHARD False False 2647905280
1b 65536 8 a100_80gb 46.18 61.57 144 1 2 16 0 24353 3044 1048576 amp_bf16 DEFAULT FULL_SHARD True False 1445974016
1b 32768 8 a100_80gb 55.52 55.52 173 1 4 32 1 50207 6275 1048576 amp_bf16 DEFAULT FULL_SHARD False False 1378865152
1b 16384 8 a100_80gb 56.6 56.6 176 2 4 64 4 79650 9956 1048576 amp_bf16 DEFAULT FULL_SHARD False False 1345310720
1b 8192 8 a100_80gb 56.69 56.69 176 3 4 96 13 110516 13814 786432 amp_bf16 DEFAULT FULL_SHARD False False 1328533504
1b 4096 8 a100_80gb 59.0 59.0 184 7 4 224 34 142457 17807 917504 amp_bf16 DEFAULT FULL_SHARD False False 1320144896
1b 2048 8 a100_80gb 59.86 59.86 186 14 4 448 80 164109 20513 917504 amp_bf16 DEFAULT FULL_SHARD False False 1315950592
1b 1024 8 a100_80gb 60.15 60.15 187 18 4 576 172 176898 22112 589824 amp_bf16 DEFAULT FULL_SHARD False False 1313853440
1b 512 8 a100_80gb 60.68 60.68 189 56 4 1792 361 185186 23148 917504 amp_bf16 DEFAULT FULL_SHARD False False 1312804864
760m 65536 8 a100_80gb 45.34 60.45 141 1 2 16 0 33150 4143 1048576 amp_bf16 DEFAULT FULL_SHARD True False 857988096
760m 32768 8 a100_80gb 54.57 54.57 170 1 2 16 2 70417 8802 524288 amp_bf16 DEFAULT FULL_SHARD False False 807656448
760m 16384 8 a100_80gb 54.64 54.64 170 3 2 48 6 114198 14274 786432 amp_bf16 DEFAULT FULL_SHARD False False 782490624
760m 8192 8 a100_80gb 55.31 55.31 172 6 2 96 20 167471 20933 786432 amp_bf16 DEFAULT FULL_SHARD False False 769907712
760m 4096 8 a100_80gb 56.05 56.05 174 12 2 192 53 218808 27351 786432 amp_bf16 DEFAULT FULL_SHARD False False 763616256
760m 2048 8 a100_80gb 56.85 56.85 177 24 2 384 126 259472 32434 786432 amp_bf16 DEFAULT FULL_SHARD False False 760470528
760m 1024 8 a100_80gb 47.76 47.76 149 48 2 768 232 238122 29765 786432 amp_bf16 DEFAULT FULL_SHARD False False 758897664
760m 512 8 a100_80gb 45.07 45.07 140 96 2 1536 460 235571 29446 786432 amp_bf16 DEFAULT FULL_SHARD False False 758111232
350m 65536 8 a100_80gb 52.7 52.7 164 1 2 16 0 60195 7524 1048576 amp_bf16 DEFAULT FULL_SHARD False False 420997120
350m 32768 8 a100_80gb 52.46 52.46 163 2 2 32 3 109222 13652 1048576 amp_bf16 DEFAULT FULL_SHARD False False 387442688
350m 16384 8 a100_80gb 53.28 53.28 166 4 2 64 11 188478 23559 1048576 amp_bf16 DEFAULT FULL_SHARD False False 370665472
350m 8192 8 a100_80gb 53.8 53.8 167 8 2 128 35 292559 36569 1048576 amp_bf16 DEFAULT FULL_SHARD False False 362276864
350m 4096 8 a100_80gb 53.31 53.31 166 16 2 256 96 396442 49555 1048576 amp_bf16 DEFAULT FULL_SHARD False False 358082560
350m 2048 8 a100_80gb 51.62 51.62 161 32 2 512 229 470263 58782 1048576 amp_bf16 DEFAULT FULL_SHARD False False 355985408
350m 1024 8 a100_80gb 50.51 50.51 157 64 2 1024 506 518504 64813 1048576 amp_bf16 DEFAULT FULL_SHARD False False 354936832
350m 512 8 a100_80gb 50.61 50.61 157 128 2 2048 1083 554643 69330 1048576 amp_bf16 DEFAULT FULL_SHARD False False 354412544
125m 65536 8 a100_80gb 54.13 54.13 168 1 2 16 2 162946 20368 1048576 amp_bf16 DEFAULT FULL_SHARD False False 174070272
125m 32768 8 a100_80gb 52.71 52.71 164 2 2 32 8 291256 36407 1048576 amp_bf16 DEFAULT FULL_SHARD False False 148904448
125m 16384 8 a100_80gb 50.61 50.61 157 4 2 64 29 480322 60040 1048576 amp_bf16 DEFAULT FULL_SHARD False False 136321536
125m 8192 8 a100_80gb 48.85 48.85 152 8 2 128 88 723142 90392 1048576 amp_bf16 DEFAULT FULL_SHARD False False 130030080
125m 4096 8 a100_80gb 46.08 46.08 143 16 2 256 231 947172 118396 1048576 amp_bf16 DEFAULT FULL_SHARD False False 126884352
125m 2048 8 a100_80gb 44.79 44.79 139 40 2 640 557 1142641 142830 1310720 amp_bf16 DEFAULT FULL_SHARD False False 125311488
125m 2048 8 a100_80gb 44.45 44.45 138 32 2 512 553 1133901 141737 1048576 amp_bf16 DEFAULT FULL_SHARD False False 125311488
125m 1024 8 a100_80gb 43.15 43.15 134 64 2 1024 1222 1251751 156468 1048576 amp_bf16 DEFAULT FULL_SHARD False False 124525056
125m 512 8 a100_80gb 42.56 42.56 132 128 2 2048 2588 1325455 165681 1048576 amp_bf16 DEFAULT FULL_SHARD False False 124131840

A100 40GB with 1600 Gbps node-node interconnect (RoCE)

Model SeqLen (T) # GPUs GPU MFU HFU Model TFLOP MicroBatchSize GradAccum GlobalBatchSize Throughput (S/s) Throughput (T/s) Throughput (T/s/GPU) GlobalBatchSize (T) Precision MP Mode Sharding Strategy Activation Checkpointing Activation CPUOffload NumParams
70b 2048 128 a100_40gb 48.91 65.21 152 4 1 512 23 48194 376 1048576 bf16 PURE FULL_SHARD True False 64862437376
70b 2048 64 a100_40gb 35.87 47.82 111 2 1 128 8 17672 276 262144 bf16 PURE FULL_SHARD True False 64862437376
30b 2048 128 a100_40gb 52.25 69.66 163 6 1 768 54 110803 865 1572864 bf16 PURE FULL_SHARD True False 29975214080
30b 2048 32 a100_40gb 51.74 68.98 161 4 1 128 13 27431 857 262144 bf16 PURE FULL_SHARD True False 29975214080
13b 8192 8 a100_40gb 43.95 58.6 137 1 16 128 1 11258 1407 1048576 bf16 PURE FULL_SHARD True False 12885411840
13b 4096 8 a100_40gb 44.85 59.8 139 2 16 256 3 12830 1603 1048576 bf16 PURE FULL_SHARD True False 12864440320
13b 2048 128 a100_40gb 51.93 69.24 162 16 1 2048 123 252444 1972 4194304 bf16 PURE FULL_SHARD True False 12853954560
13b 2048 64 a100_40gb 52.04 69.39 162 16 1 1024 61 126479 1976 2097152 bf16 PURE FULL_SHARD True False 12853954560
13b 2048 32 a100_40gb 52.62 70.16 164 14 1 448 31 63946 1998 917504 bf16 PURE FULL_SHARD True False 12853954560
13b 2048 16 a100_40gb 52.5 70.0 163 10 1 160 15 31900 1993 327680 bf16 PURE FULL_SHARD True False 12853954560
13b 2048 8 a100_40gb 43.94 58.58 137 4 16 512 6 13347 1668 1048576 bf16 PURE FULL_SHARD True False 12853954560
13b 1024 8 a100_40gb 44.07 58.76 137 8 16 1024 13 13817 1727 1048576 bf16 PURE FULL_SHARD True False 12848711680
13b 512 8 a100_40gb 44.28 59.04 138 16 16 2048 27 14108 1763 1048576 bf16 PURE FULL_SHARD True False 12846090240
7b 16384 8 a100_40gb 47.65 63.53 148 1 4 32 1 17998 2249 524288 bf16 PURE FULL_SHARD True False 6717579264
7b 8192 8 a100_40gb 49.04 65.38 153 3 4 96 2 23098 2887 786432 bf16 PURE FULL_SHARD True False 6684024832
7b 4096 8 a100_40gb 50.11 66.82 156 6 4 192 6 26930 3366 786432 bf16 PURE FULL_SHARD True False 6667247616
7b 2048 128 a100_40gb 50.14 66.85 156 18 1 2304 226 463749 3623 4718592 bf16 PURE FULL_SHARD True False 6658859008
7b 2048 64 a100_40gb 50.73 67.64 158 18 1 1152 114 234614 3665 2359296 bf16 PURE FULL_SHARD True False 6658859008
7b 2048 32 a100_40gb 51.55 68.73 160 18 1 576 58 119202 3725 1179648 bf16 PURE FULL_SHARD True False 6658859008
7b 2048 16 a100_40gb 50.44 67.26 157 16 1 256 28 58322 3645 524288 bf16 PURE FULL_SHARD True False 6658859008
7b 2048 8 a100_40gb 50.92 67.89 158 12 4 384 14 29436 3679 786432 bf16 PURE FULL_SHARD True False 6658859008
7b 1024 8 a100_40gb 51.31 68.42 160 24 4 768 30 30833 3854 786432 bf16 PURE FULL_SHARD True False 6654664704
7b 512 8 a100_40gb 50.85 67.8 158 48 4 1536 60 31167 3895 786432 bf16 PURE FULL_SHARD True False 6652567552
3b 32768 8 a100_40gb 46.03 61.37 143 1 4 32 0 23640 2955 1048576 bf16 PURE FULL_SHARD True False 2730480640
3b 16384 8 a100_40gb 46.14 61.52 143 2 8 128 2 35726 4465 2097152 bf16 PURE FULL_SHARD True False 2688537600
3b 8192 8 a100_40gb 55.13 55.13 172 1 8 64 6 57193 7149 524288 bf16 PURE FULL_SHARD False False 2667566080
3b 4096 8 a100_40gb 56.18 56.18 175 2 8 128 17 70223 8777 524288 bf16 PURE FULL_SHARD False False 2657080320
3b 2048 128 a100_40gb 54.8 54.8 170 6 1 768 596 1220885 9538 1572864 bf16 PURE FULL_SHARD False False 2651837440
3b 2048 64 a100_40gb 55.94 55.94 174 6 1 384 304 623167 9736 786432 bf16 PURE FULL_SHARD False False 2651837440
3b 2048 32 a100_40gb 56.96 56.96 177 6 1 192 154 317261 9914 393216 bf16 PURE FULL_SHARD False False 2651837440
3b 2048 16 a100_40gb 56.02 56.02 174 5 1 80 76 156013 9750 163840 bf16 PURE FULL_SHARD False False 2651837440
3b 2048 8 a100_40gb 57.82 57.82 180 5 8 320 39 80520 10065 655360 bf16 PURE FULL_SHARD False False 2651837440
3b 1024 8 a100_40gb 58.14 58.14 181 10 8 640 83 85854 10731 655360 bf16 PURE FULL_SHARD False False 2649216000
3b 512 8 a100_40gb 59.49 59.49 185 20 8 1280 176 90596 11324 655360 bf16 PURE FULL_SHARD False False 2647905280
1b 32768 8 a100_40gb 45.07 60.1 140 1 4 32 1 40762 5095 1048576 bf16 PURE FULL_SHARD True False 1378865152
1b 16384 8 a100_40gb 55.23 55.23 172 1 8 64 4 77723 9715 1048576 bf16 PURE FULL_SHARD False False 1345310720
1b 8192 8 a100_40gb 55.29 55.29 172 2 8 128 13 107799 13474 1048576 bf16 PURE FULL_SHARD False False 1328533504
1b 4096 8 a100_40gb 55.85 55.85 174 4 8 256 32 134851 16856 1048576 bf16 PURE FULL_SHARD False False 1320144896
1b 2048 128 a100_40gb 54.41 54.41 169 10 1 1280 1165 2386897 18647 2621440 bf16 PURE FULL_SHARD False False 1315950592
1b 2048 64 a100_40gb 55.44 55.44 172 10 1 640 593 1216104 19001 1310720 bf16 PURE FULL_SHARD False False 1315950592
1b 2048 32 a100_40gb 45.39 45.39 141 10 1 320 243 497782 15555 655360 bf16 PURE FULL_SHARD False False 1315950592
1b 2048 16 a100_40gb 55.69 55.69 173 8 1 128 149 305372 19085 262144 bf16 PURE FULL_SHARD False False 1315950592
1b 2048 8 a100_40gb 56.23 56.23 175 8 8 512 75 154171 19271 1048576 bf16 PURE FULL_SHARD False False 1315950592
1b 1024 8 a100_40gb 57.02 57.02 177 16 8 1024 163 167677 20959 1048576 bf16 PURE FULL_SHARD False False 1313853440
1b 512 8 a100_40gb 57.1 57.1 178 32 8 2048 340 174256 21782 1048576 bf16 PURE FULL_SHARD False False 1312804864
760m 32768 8 a100_40gb 44.53 59.37 138 1 4 32 1 57464 7183 1048576 bf16 PURE FULL_SHARD True False 807656448
760m 16384 8 a100_40gb 53.26 53.26 166 1 4 32 6 111316 13914 524288 bf16 PURE FULL_SHARD False False 782490624
760m 8192 8 a100_40gb 53.12 53.12 165 3 4 96 19 160853 20106 786432 bf16 PURE FULL_SHARD False False 769907712
760m 4096 8 a100_40gb 53.0 53.0 165 6 4 192 50 206909 25863 786432 bf16 PURE FULL_SHARD False False 763616256
760m 2048 128 a100_40gb 50.73 50.73 158 12 1 1536 1808 3704382 28940 3145728 bf16 PURE FULL_SHARD False False 760470528
760m 2048 64 a100_40gb 51.44 51.44 160 12 1 768 917 1878030 29344 1572864 bf16 PURE FULL_SHARD False False 760470528
760m 2048 32 a100_40gb 51.97 51.97 162 12 1 384 463 948745 29648 786432 bf16 PURE FULL_SHARD False False 760470528
760m 2048 16 a100_40gb 51.9 51.9 161 12 1 192 231 473723 29607 393216 bf16 PURE FULL_SHARD False False 760470528
760m 2048 8 a100_40gb 52.89 52.89 165 12 4 384 117 241389 30173 786432 bf16 PURE FULL_SHARD False False 760470528
760m 1024 8 a100_40gb 53.63 53.63 167 24 4 768 261 267376 33422 786432 bf16 PURE FULL_SHARD False False 758897664
760m 512 8 a100_40gb 53.47 53.47 166 48 4 1536 545 279504 34938 786432 bf16 PURE FULL_SHARD False False 758111232
350m 32768 8 a100_40gb 51.55 51.55 160 1 4 32 3 107329 13416 1048576 bf16 PURE FULL_SHARD False False 387442688
350m 16384 8 a100_40gb 51.78 51.78 161 2 4 64 11 183175 22896 1048576 bf16 PURE FULL_SHARD False False 370665472
350m 8192 8 a100_40gb 51.39 51.39 160 4 4 128 34 279466 34933 1048576 bf16 PURE FULL_SHARD False False 362276864
350m 4096 8 a100_40gb 50.38 50.38 157 8 4 256 91 374670 46833 1048576 bf16 PURE FULL_SHARD False False 358082560
350m 2048 128 a100_40gb 45.61 45.61 142 18 1 2304 3245 6647647 51934 4718592 bf16 PURE FULL_SHARD False False 355985408
350m 2048 64 a100_40gb 46.27 46.27 144 18 1 1152 1646 3372118 52689 2359296 bf16 PURE FULL_SHARD False False 355985408
350m 2048 32 a100_40gb 47.26 47.26 147 18 1 576 840 1721978 53811 1179648 bf16 PURE FULL_SHARD False False 355985408
350m 2048 16 a100_40gb 48.66 48.66 151 18 1 288 432 886622 55413 589824 bf16 PURE FULL_SHARD False False 355985408
350m 2048 8 a100_40gb 49.17 49.17 153 16 4 512 218 447963 55995 1048576 bf16 PURE FULL_SHARD False False 355985408
350m 1024 8 a100_40gb 48.73 48.73 152 32 4 1024 488 500184 62523 1048576 bf16 PURE FULL_SHARD False False 354936832
350m 512 8 a100_40gb 48.39 48.39 150 64 4 2048 1035 530277 66284 1048576 bf16 PURE FULL_SHARD False False 354412544
125m 32768 8 a100_40gb 47.27 47.27 147 1 4 32 7 261208 32651 1048576 bf16 PURE FULL_SHARD False False 148904448
125m 16384 8 a100_40gb 46.77 46.77 145 2 3 48 27 443876 55484 786432 bf16 PURE FULL_SHARD False False 136321536
125m 8192 8 a100_40gb 46.94 46.94 146 5 3 120 84 694868 86858 983040 bf16 PURE FULL_SHARD False False 130030080
125m 4096 8 a100_40gb 44.82 44.82 139 13 3 312 224 921297 115162 1277952 bf16 PURE FULL_SHARD False False 126884352
125m 2048 128 a100_40gb 38.86 38.86 121 26 1 3328 7746 15863837 123936 6815744 bf16 PURE FULL_SHARD False False 125311488
125m 2048 64 a100_40gb 39.27 39.27 122 26 1 1664 3913 8015010 125234 3407872 bf16 PURE FULL_SHARD False False 125311488
125m 2048 32 a100_40gb 39.86 39.86 124 26 1 832 1986 4067922 127122 1703936 bf16 PURE FULL_SHARD False False 125311488
125m 2048 16 a100_40gb 40.93 40.93 127 26 1 416 1019 2088560 130535 851968 bf16 PURE FULL_SHARD False False 125311488
125m 2048 8 a100_40gb 42.75 42.75 133 26 3 624 532 1090678 136334 1277952 bf16 PURE FULL_SHARD False False 125311488
125m 1024 8 a100_40gb 40.89 40.89 127 52 3 1248 1158 1186314 148289 1277952 bf16 PURE FULL_SHARD False False 124525056
125m 512 8 a100_40gb 40.26 40.26 125 104 3 2496 2448 1253886 156735 1277952 bf16 PURE FULL_SHARD False False 124131840