Speed up CUDA build by splitting SM80 MoE GEMM generated files

[tianleiwu]

Description

The SM80 fused MoE GEMM launchers were generated as two monolithic translation units — fused_moe_gemm_sm80_bf16.generated.cu and fused_moe_gemm_sm80_f16.generated.cu — each packing 30 CUTLASS kernel instantiations (5 tile shapes × 3 stages × 2 epilogues) into a single .cu file. nvcc's --threads flag only parallelizes across GPU architectures (80;90) within a file; it does not parallelize template instantiations inside one translation unit. As a result each of these two objects was a long, serial compile on the build's critical path (each ~5–6 min in a Debug 80;90 build), and Ninja could only overlap the two of them.

This PR updates the kernel generator to split the SM80 instantiations by tile shape, producing 10 smaller files (5 tile shapes × 2 dtypes, 6 kernels each) that Ninja compiles in parallel.

Key Changes

• generate_moe_kernels.py: the SM80 loop now groups instantiations by (element_type, tile_shape) and emits one file per group, named fused_moe_gemm_sm80_{dtype}_m{M}_n{N}_k{K}.generated.cu. Added glob-based cleanup that removes stale SM80 generated files (the old monolithic per-dtype files and any files from a previous tile configuration) so they are not compiled.
• Regenerated the launcher directory: removed the 2 monolithic files and added the 10 split files. Total number of compiled kernel instantiations is unchanged.

cmake already discovers these via GLOB_RECURSE ... CONFIGURE_DEPENDS, so no cmake changes are needed — the new files are picked up automatically on the next configure.

Testing

• Verified generator idempotency (re-running reports all files up to date) and stale-file cleanup.
• Built the new objects in a Debug CMAKE_CUDA_ARCHITECTURES="80;90" build:
  • one 6-kernel split object compiles in ~67s;
  • all 10 split objects build in ~182s wall-clock in parallel.
• Previously the two 30-kernel objects were ~5× the per-file cost each and only 2-way parallel, so this substantially shortens the MoE portion of the critical path on multi-core machines. Total compiled kernels (and thus runtime coverage) is unchanged.

To regenerate after future tile-shape changes:

python onnxruntime/contrib_ops/cuda/llm/generate_moe_kernels.py -a "80;90" -o onnxruntime/contrib_ops/cuda/llm/moe_gemm/launchers