[Inductor cutlass backend] disable StreamK for GEMMs due to performance issues

[kadeng]

Stack from ghstack (oldest at bottom):

• [Inductor cutlass backend] Fixed workspace resize issue #115877
• [Inductor cutlass backend] Log CUDA compilation times #115814
• [Inductor cutlass backend] Added (some) doc comments and type annotations #115813
• [Inductor cutlass backend] Test and fix broadcasting of Aux / Bias edge cases #115655
• [Inductor cutlass backend] Robust Precompilation / Autotuning / Retuning in subprocesses #115654
• [Cutlass 3.3 submodule upgrade] #112861
• [Inductor cutlass backend] Support for StreamK and non-zero workspace sizes #114687
• [Inductor cutlass backend] Add support for row/column broadcasting of auxiliary inputs #115270
• [Inductor cutlass backend] Provide CUDA SM count to kernels #115174
• [Inductor cutlass backend] Generate standalone debug runners for Cutlass GEMMs #115072
• [inductor max autotune] Support sigmoid and tanh activations #114606
• [inductor max autotune] Flexible GEMM layout autotuning #114319
• [Inductor cutlass backend] Allow to fall back to non-fuseable ops #114075
• [Inductor cutlass backend] Experimental extended Cutlass op generator #113932
• [inductor max autotune] Log but continue on errors during autotuning benchmark #113891
• [Inductor Cutlass backend] Enabling flexible EVT-based pointwise fusions with additional tensor input #113959
• [Inductor cutlass backend] Enable torch.bmm and torch.baddbmm support #113890
• -> [Inductor cutlass backend] disable StreamK for GEMMs due to performance issues #113570
• [Cutlass inductor backend] Cutlass GEMM size threshold #113569
• [Inductor max autotune] Multithreaded Precompilation #113558
• [inductor max autotune] Detailed autotuning result logs ( machine-readable ) #113399
• [Inductor cutlass GEMM backend] Fix for operand memory layout change between autotuning and CUTLASSGEMMTemplate.render #113366
• [Inductor] Fix debug_str method of FusedSchedulerNode #113365

We observed some Cutlass GEMM ops with StreamK enabled to take forever during autotuning. Disabling them for now to unblock.

We should investigate this later.

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @muchulee8 @aakhundov @ColinPeppler

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/113570

• 📄 Preview Python docs built from this PR
• 📄 Preview C++ docs built from this PR
• ❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 13 New Failures, 13 Unrelated Failures

As of commit 906adc2 with merge base afe6d27 ():

NEW FAILURES - The following jobs have failed:

• pull / linux-docs / build-docs-python-false (gh)
  Process completed with exit code 2.
• pull / linux-focal-cuda12.1-py3.10-gcc9-sm86 / test (default, 1, 5, linux.g5.4xlarge.nvidia.gpu) (gh)
  inductor/test_aot_inductor.py::AOTInductorTestNonABICompatibleCuda::test_zero_grid_with_unbacked_symbols_non_abi_compatible_cuda
• pull / linux-focal-cuda12.1-py3.10-gcc9-sm86 / test (default, 2, 5, linux.g5.4xlarge.nvidia.gpu) (gh)
  inductor/test_unbacked_symints.py::TestUnbackedSymints::test_autotuning
• pull / linux-focal-py3.8-clang10 / test (crossref, 1, 2, linux.2xlarge) (gh)
  RuntimeError: inductor/test_debug_trace 1/1 failed
• pull / linux-focal-py3.8-clang10 / test (crossref, 2, 2, linux.2xlarge) (gh)
  RuntimeError: inductor/test_torchinductor_dynamic_shapes 1/1 failed
• pull / linux-focal-py3.8-clang10 / test (default, 1, 3, linux.2xlarge) (gh)
  RuntimeError: inductor/test_debug_trace 1/1 failed
• pull / linux-focal-py3.8-clang10 / test (default, 2, 3, linux.2xlarge) (gh)
  RuntimeError: inductor/test_max_autotune 1/1 failed
• pull / linux-focal-py3.8-clang10 / test (default, 3, 3, linux.2xlarge) (gh)
  RuntimeError: inductor/test_torchinductor 2/2 failed
• pull / linux-jammy-py3.8-gcc11 / test (default, 1, 3, linux.2xlarge) (gh)
  RuntimeError: inductor/test_debug_trace 1/1 failed
• pull / linux-jammy-py3.8-gcc11 / test (default, 2, 3, linux.2xlarge) (gh)
  RuntimeError: inductor/test_max_autotune 1/1 failed
• pull / linux-jammy-py3.8-gcc11 / test (default, 3, 3, linux.2xlarge) (gh)
  RuntimeError: inductor/test_torchinductor 2/2 failed
• pull / linux-jammy-py3.8-gcc11 / test (distributed, 1, 2, linux.2xlarge) (gh)
  RuntimeError: distributed/test_inductor_collectives 1/1 failed
• pull / linux-jammy-py3.8-gcc11 / test (distributed, 2, 2, linux.2xlarge) (gh)
  RuntimeError: distributed/_spmd/test_transformation 1/1 failed

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

• inductor / linux-jammy-cpu-py3.8-gcc11-inductor / test (cpu_inductor_huggingface, 1, 1, linux.12xlarge) (gh)
  YituTechConvBert
• inductor / linux-jammy-cpu-py3.8-gcc11-inductor / test (cpu_inductor_timm, 1, 2, linux.12xlarge) (gh)
  mixer_b16_224
• inductor / linux-jammy-cpu-py3.8-gcc11-inductor / test (cpu_inductor_timm, 2, 2, linux.12xlarge) (gh)
  xcit_large_24_p8_224
• inductor / linux-jammy-cpu-py3.8-gcc11-inductor / test (cpu_inductor_torchbench, 1, 2, linux.12xlarge) (gh)
  hf_T5_base
• inductor / linux-jammy-cpu-py3.8-gcc11-inductor / test (cpu_inductor_torchbench, 2, 2, linux.12xlarge) (gh)
  yolov3
• inductor / linux-jammy-cpu-py3.8-gcc11-inductor / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.12xlarge) (gh)
  YituTechConvBert
• inductor / linux-jammy-cpu-py3.8-gcc11-inductor / test (dynamic_cpu_inductor_timm, 1, 2, linux.12xlarge) (gh)
  mixer_b16_224
• inductor / linux-jammy-cpu-py3.8-gcc11-inductor / test (dynamic_cpu_inductor_timm, 2, 2, linux.12xlarge) (gh)
  xcit_large_24_p8_224
• inductor / linux-jammy-cpu-py3.8-gcc11-inductor / test (dynamic_cpu_inductor_torchbench, 1, 2, linux.12xlarge) (gh)
  hf_T5_base
• inductor / linux-jammy-cpu-py3.8-gcc11-inductor / test (dynamic_cpu_inductor_torchbench, 2, 2, linux.12xlarge) (gh)
  yolov3
• pull / linux-focal-py3_8-clang9-xla / test (xla, 1, 1, linux.12xlarge) (gh)
  ModuleNotFoundError: No module named 'torch.version'
• pull / linux-focal-py3.8-clang10 / test (dynamo, 1, 2, linux.2xlarge) (gh)
  Process completed with exit code 1.
• pull / linux-focal-py3.8-clang10 / test (dynamo, 2, 2, linux.2xlarge) (gh)
  Process completed with exit code 1.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[kadeng]

Moved to #115919