Added prefill strategy benchmarking script and results

[jijiaz]

MoE Prefill Optimization Strategy Benchmarking

We refreshed the benchmarking results using the updated optimized kernel and expanded the reported metrics to improve data completeness. In addition to latency and speedup, the updated benchmark now also reports baseline and hybrid TFLOPs.

The benchmark script now covers both threshold tuning and API-path measurement:

Running --mode full evaluates thresholds [64, 128, 256, 512, 1024], exposing the tuning process for the threshold parameter across all tested dtypes/formats.
Running --mode summary reports a compact table at the script's default threshold, which is currently 256.
The script also supports two API modes:
--api wrapper: measures the public Python ARK API, including wrapper-side validation, normalization, and output/workspace allocation overhead.
--api raw: calls ark.xpu_lib directly with preallocated buffers, which does not reimplement the kernel math, but bypasses Python wrapper overhead so the result more closely reflects the underlying kernel plus launch/runtime cost.

This distinction matters for interpretation: wrapper results represent end-to-end public API behavior, while raw results are better suited for strategy studies that want to minimize machine-specific wrapper overhead (for example, scalar-sync cost from num_tokens.sum().item()).

Based on the current full sweep results, 256 is the best-performing threshold across all tested dtypes/formats, while larger thresholds such as 1024 can become counterproductive.

Key Findings:

• Optimal threshold identification: 256-token threshold consistently delivers the best performance across all tested dtypes/formats
• Threshold tuning process added: --mode full exposes the full threshold sweep across [64, 128, 256, 512, 1024]
• Stronger speedups with the updated kernel: up to 2.07x for FP16 and 2.10x for BF16, with quantized/FP8 formats improving by roughly 1.16x-1.43x
• More complete reporting: benchmark results now include baseline_ms, hybrid_ms, speedup, base_TFLOPs, and hybrid_TFLOPs
• Format-agnostic benefits: performance gains remain consistent across FP16/BF16, INT8/INT4/INT2 (sym/asym), and FP8 (E4M3/E5M2)
• Dual measurement semantics: wrapper reflects end-to-end public API cost, while raw isolates kernel/runtime-facing behavior for cleaner baseline-vs-hybrid strategy comparison

Performance Summary (best threshold = 256):

Format	Dtype	Baseline (ms)	Optimized (ms)	Speedup	Base TFLOPs	Hybrid TFLOPs	Threshold
float16	float16	26.42	12.75	2.07x	11.23	57.29	256
bfloat16	bfloat16	26.06	12.42	2.10x	11.39	58.81	256
int8_sym	bfloat16	51.09	43.68	1.17x	5.81	16.72	256
int8_asym	bfloat16	54.11	46.75	1.16x	5.48	15.62	256
int4_sym	bfloat16	48.15	39.19	1.23x	6.16	18.64	256
int4_asym	bfloat16	51.41	43.23	1.19x	5.77	16.89	256
int2_sym	bfloat16	37.28	26.12	1.43x	7.96	27.96	256
int2_asym	bfloat16	39.25	28.39	1.38x	7.56	25.72	256
fp8_e4m3	bfloat16	52.57	45.10	1.17x	5.64	16.19	256
fp8_e5m2	bfloat16	52.16	44.42	1.17x	5.69	16.44	256

The results show that with the updated kernel, the hybrid pad-and-clip strategy delivers materially better gains than before, especially for FP16/BF16, while still providing stable improvements across quantized and FP8 formats.

Benchmark Configuration: 256 experts, 31,440 routed tokens, 238 active experts, N=3072, K=1536, group_size=128.