metal : tuning Flash-Attention kernel across M-chips

[ggerganov]

Overview

Companion thread to #26570. The ggml-metal-tuning fa-vec tuner sweeps the flash-attn vec (Q, NE) config grid on the machine it runs on and prints pasteable rows for fa_vec_tuned_table. If your Apple GPU is not checked off below, a sweep from your machine would provide the optimal FA parameters to be included in the Metal backend for better performance on this GPU family.

Running the tuning

On the target Mac, build from source:

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp

cmake -B build -DGGML_METAL=ON
cmake --build build --target ggml-metal-tuning -j

Run the following command:

./build/bin/ggml-metal-tuning fa-vec --dtype f16,q8_0 > fa_vec_rows.txt 2> fa_vec_sweep.log

Important

A full sweep takes a few hours, so it is best to run the tuning over night while the machine is not being used

Reporting results

Post in this thread (or open a PR) with:

• the contents of fa_vec_rows.txt (rows only, ready to paste into fa_vec_tuned_table)

Devices collected so far

• M1
• M1 Pro
• M1 Max
• M1 Ultra
• M2
• M2 Pro
• M2 Max
• M2 Ultra
• M3
• M3 Pro
• M3 Max
• M3 Ultra
• M4
• M4 Pro
• M4 Max
• M5
• M5 Pro
• M5 Max
• M5 Ultra

[jruhe-adesso]

So there is no need to differentiate between SoC-sub-types, e.g. M5 Pro with 16 GPU Core vs 20 GPU Cores?

[ggerganov]

Not sure - keeping it simple for now.