perf2

System Configuration

• Sockets: 2
• CPU: Intel Xeon Gold 5418Y CPU / 24 cores per socket / 48 cores total / SMT-enabled
• Memory: 256 GB / DDR5 / 4800 MT/s

Intel AMX Performance Benchmark

2D Tile Registers

The 8 2D tile registers are each 1KiB in size. These registers have 16 rows each of 64 bytes. For BF16 this means, a matrix of 16 * 32 BF16 numbers whereas for INT8, this means a matrix of 16 * 64 INT8 numbers.

Configuration

1. TMUL: 48 (1 per core)
2. Tiles: 48 * 8 * 1KiB = 384 KiB (1 per TMUL)

Results

----------------------------------------------------------------------------------------
|   Mode   |    N1 / N2 / M    | Data size (MiB) | Total FLOP | Duration (us) | GFLOPS |
----------------------------------------------------------------------------------------
| IP / AMX | 64/64/64          |        0.016384 |     520192 |            90 | 5.77991 |
| IP / AMX | 128/128/128       |        0.065536 | 4.17792e+06 |            92 | 45.4122 |
| IP / AMX | 256/256/256       |        0.262144 | 3.34889e+07 |           134 | 249.917 |
| IP / AMX | 512/512/512       |         1.04858 | 2.68173e+08 |           123 | 2180.27 |
| IP / AMX | 1024/1024/1024    |          4.1943 | 2.14644e+09 |           497 | 4318.78 |
| IP / AMX | 2048/2048/2048    |         16.7772 | 1.71757e+10 |          1549 | 11088.2 |
| IP / AMX | 4096/4096/4096    |         67.1089 | 1.37422e+11 |          7862 | 17479.3 |
| IP / AMX | 8192/8192/8192    |         268.435 | 1.09944e+12 |         69447 | 15831.4 |
| IP / AMX | 16384/16384/16384 |         1073.74 | 8.79582e+12 |        431807 | 20369.8 |
| IP / AMX | 32768/32768/32768 |         4294.97 | 7.03677e+13 |    3.2263e+06 | 21810.6 |
----------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------
|    Mode    |    N1 / N2 / M    | Data size (MiB) | Total FLOP | Duration (us) | GFLOPS |
------------------------------------------------------------------------------------------
| GEMM / AMX | 64/64/64          |        0.016384 |     520192 |            21 | 24.771 |
| GEMM / AMX | 128/128/128       |        0.065536 | 4.17792e+06 |            27 | 154.738 |
| GEMM / AMX | 256/256/256       |        0.262144 | 3.34889e+07 |            26 | 1288.03 |
| GEMM / AMX | 512/512/512       |         1.04858 | 2.68173e+08 |            51 | 5258.3 |
| GEMM / AMX | 1024/1024/1024    |          4.1943 | 2.14644e+09 |           105 | 20442.2 |
| GEMM / AMX | 2048/2048/2048    |         16.7772 | 1.71757e+10 |           757 | 22689.1 |
| GEMM / AMX | 4096/4096/4096    |         67.1089 | 1.37422e+11 |          8158 | 16845.1 |
| GEMM / AMX | 8192/8192/8192    |         268.435 | 1.09944e+12 |         89320 | 12309.1 |
| GEMM / AMX | 16384/16384/16384 |         1073.74 | 8.79582e+12 |        884846 | 9940.51 |
| GEMM / AMX | 32768/32768/32768 |         4294.97 | 7.03677e+13 |   2.14322e+06 | 32832.7 |
-------------------------------------------------------------------------------------------

---------------------------------------------------------------------------------------
|   Mode   |    N1 / N2 / M   | Data size (MiB) | Total FLOP | Duration (us) | GFLOPS |
---------------------------------------------------------------------------------------
| IP / AMX | 1/1048576/1024   |         2147.49 | 2.14644e+09 |          2168 | 990.053 |
| IP / AMX | 2/1048576/1024   |         2147.49 | 4.29287e+09 |          3363 | 1276.5 |
| IP / AMX | 4/1048576/1024   |         2147.49 | 8.58574e+09 |          3421 | 2509.72 |
| IP / AMX | 8/1048576/1024   |          2147.5 | 1.71715e+10 |          3768 | 4557.19 |
| IP / AMX | 16/1048576/1024  |         2147.52 | 3.4343e+10 |          4141 | 8293.4 |
| IP / AMX | 32/1048576/1024  |         2147.55 | 6.86859e+10 |         20002 | 3433.95 |
| IP / AMX | 64/1048576/1024  |         2147.61 | 1.37372e+11 |         11177 | 12290.6 |
| IP / AMX | 128/1048576/1024 |         2147.75 | 2.74744e+11 |         21950 | 12516.8 |
| IP / AMX | 256/1048576/1024 |         2148.01 | 5.49487e+11 |         51648 | 10639.1 |
| IP / AMX | 512/1048576/1024   |         2148.53 | 1.09897e+12 |        115785 | 9491.51 |
| IP / AMX | 1024/1048576/1024  |         2149.58 | 2.19795e+12 |        222587 | 9874.56 |
| IP / AMX | 2048/1048576/1024  |         2151.68 | 4.3959e+12 |        424706 | 10350.5 |
| IP / AMX | 4096/1048576/1024  |         2155.87 | 8.7918e+12 |        823180 | 10680.3 |
| IP / AMX | 8192/1048576/1024  |         2164.26 | 1.75836e+13 |   1.62587e+06 | 10814.9 |
| IP / AMX | 16384/1048576/1024 |         2181.04 | 3.51672e+13 |   2.80431e+06 | 12540.4 |
| IP / AMX | 32768/1048576/1024 |         2214.59 | 7.03344e+13 |   6.01659e+06 | 11690.1 |
| IP / AMX | 16/1048576/1024 |         2048.03 | 3.4343e+10 |   3.92304e+06 | 8754.17 |
| IP / AMX | 16/2097152/1024 |         4096.03 | 6.86859e+10 |   7.75651e+06 | 8855.26 |
| IP / AMX | 16/4194304/1024 |         8192.03 | 1.37372e+11 |   3.09687e+07 | 4435.83 |
| IP / AMX | 16/8388608/1024 |           16384 | 2.74744e+11 |   3.24258e+07 | 8472.99 |
| IP / AMX | 32/1048576/1024 |         2048.06 | 6.86859e+10 |   5.29124e+06 | 12981.1 |
| IP / AMX | 32/2097152/1024 |         4096.06 | 1.37372e+11 |   1.14638e+07 | 11983.1 |
| IP / AMX | 32/4194304/1024 |         8192.06 | 2.74744e+11 |   2.37991e+07 | 11544.3 |
| IP / AMX | 32/8388608/1024 |         16384.1 | 5.49487e+11 |   4.60121e+07 | 11942.2 |
| IP / AMX | 64/1048576/1024 |         2048.12 | 1.37372e+11 |   1.10513e+07 | 12430.4 |
| IP / AMX | 64/2097152/1024 |         4096.12 | 2.74744e+11 |    2.3381e+07 | 11750.7 |
| IP / AMX | 64/4194304/1024 |         8192.12 | 5.49487e+11 |   4.70061e+07 | 11689.7 |
| IP / AMX | 64/8388608/1024 |         16384.1 | 1.09897e+12 |   9.37894e+07 | 11717.5 |
| IP / AMX | 8192/1048576/256 |             516 | 4.38946e+12 |    1.8612e+09 | 2358.4 |
| IP / AMX | 10000/1000000/200 |         385.284 |   3.99e+12 |   2.44864e+09 | 1629.48 |
-----------------------------------------------------------------------------------------