[Inductor cutlass backend] Fixed workspace resize issue

[kadeng]

Stack from ghstack (oldest at bottom):

• -> [Inductor cutlass backend] Fixed workspace resize issue #115877
• [Inductor cutlass backend] Log CUDA compilation times #115814
• [Inductor cutlass backend] Added (some) doc comments and type annotations #115813
• [Inductor cutlass backend] Test and fix broadcasting of Aux / Bias edge cases #115655
• [Inductor cutlass backend] Robust Precompilation / Autotuning / Retuning in subprocesses #115654
• [Cutlass 3.3 submodule upgrade] #112861
• [Inductor cutlass backend] Support for StreamK and non-zero workspace sizes #114687
• [Inductor cutlass backend] Add support for row/column broadcasting of auxiliary inputs #115270
• [Inductor cutlass backend] Provide CUDA SM count to kernels #115174
• [Inductor cutlass backend] Generate standalone debug runners for Cutlass GEMMs #115072
• [inductor max autotune] Support sigmoid and tanh activations #114606
• [inductor max autotune] Flexible GEMM layout autotuning #114319
• [Inductor cutlass backend] Allow to fall back to non-fuseable ops #114075
• [Inductor cutlass backend] Experimental extended Cutlass op generator #113932
• [inductor max autotune] Log but continue on errors during autotuning benchmark #113891
• [Inductor Cutlass backend] Enabling flexible EVT-based pointwise fusions with additional tensor input #113959
• [Inductor cutlass backend] Enable torch.bmm and torch.baddbmm support #113890
• [Inductor cutlass backend] disable StreamK for GEMMs due to performance issues #113570
• [Cutlass inductor backend] Cutlass GEMM size threshold #113569
• [Inductor max autotune] Multithreaded Precompilation #113558
• [inductor max autotune] Detailed autotuning result logs ( machine-readable ) #113399
• [Inductor cutlass GEMM backend] Fix for operand memory layout change between autotuning and CUTLASSGEMMTemplate.render #113366
• [Inductor] Fix debug_str method of FusedSchedulerNode #113365

When the workspace size is changed via retuning, the corresponding buffer allocation in the wrapper needs to use the updated size. This did
not work properly before, this diff fixes that.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/115877

• 📄 Preview Python docs built from this PR
• 📄 Preview C++ docs built from this PR
• ❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 22 New Failures, 11 Unrelated Failures

As of commit 0ad3f3f with merge base afe6d27 ():

NEW FAILURES - The following jobs have failed:

• inductor / linux-jammy-cpu-py3.8-gcc11-inductor / test (cpu_inductor_torchbench, 1, 2, linux.12xlarge) (gh)
  hf_T5_base
• inductor / linux-jammy-cpu-py3.8-gcc11-inductor / test (dynamic_cpu_inductor_timm, 1, 2, linux.12xlarge) (gh)
  mixer_b16_224
• Lint / lintrunner / linux-job (gh)
  >>> Lint for torch/_inductor/scheduler.py:
• pull / linux-docs / build-docs-python-false (gh)
  Process completed with exit code 2.
• pull / linux-focal-cuda12.1-py3.10-gcc9 / test (default, 5, 5, linux.4xlarge.nvidia.gpu) (gh)
  test_public_bindings.py::TestPublicBindings::test_modules_can_be_imported
• pull / linux-focal-cuda12.1-py3.10-gcc9-sm86 / test (default, 1, 5, linux.g5.4xlarge.nvidia.gpu) (gh)
  inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpuWithStackAllocationAndMinimalArrayRefInterface::test_addmm_multiple_dynamic_abi_compatible_cpu_with_stack_allocation_and_minimal_arrayref_interface
• pull / linux-focal-cuda12.1-py3.10-gcc9-sm86 / test (default, 2, 5, linux.g5.4xlarge.nvidia.gpu) (gh)
  inductor/test_max_autotune.py::TestMaxAutotune::test_max_autotune_cutlass_backend_addmm_dynamic_False_max_autotune_gemm_backends_CUTLASS_cutlass_prefer_evt_capable_ops_False
• pull / linux-focal-cuda12.1-py3.10-gcc9-sm86 / test (default, 3, 5, linux.g5.4xlarge.nvidia.gpu) (gh)
  inductor/test_select_algorithm.py::TestSelectAlgorithm::test_mm_dup_args
• pull / linux-focal-cuda12.1-py3.10-gcc9-sm86 / test (default, 5, 5, linux.g5.4xlarge.nvidia.gpu) (gh)
  inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_dynamic_smem_above_default_limit_abi_compatible_cpu
• pull / linux-focal-py3.11-clang10 / test (crossref, 2, 2, linux.2xlarge) (gh)
  test_public_bindings.py::TestPublicBindings::test_modules_can_be_imported
• pull / linux-focal-py3.11-clang10 / test (default, 1, 3, linux.2xlarge) (gh)
  test_public_bindings.py::TestPublicBindings::test_modules_can_be_imported
• pull / linux-focal-py3.8-clang10 / test (crossref, 1, 2, linux.2xlarge) (gh)
  RuntimeError: inductor/test_debug_trace 1/1 failed
• pull / linux-focal-py3.8-clang10 / test (crossref, 2, 2, linux.2xlarge) (gh)
  RuntimeError: inductor/test_torchinductor 1/1 failed
• pull / linux-focal-py3.8-clang10 / test (default, 1, 3, linux.2xlarge) (gh)
  RuntimeError: inductor/test_debug_trace 1/1 failed
• pull / linux-focal-py3.8-clang10 / test (default, 2, 3, linux.2xlarge) (gh)
  RuntimeError: inductor/test_max_autotune 1/1 failed
• pull / linux-focal-py3.8-clang10 / test (default, 3, 3, linux.2xlarge) (gh)
  RuntimeError: inductor/test_torchinductor_dynamic_shapes 2/2 failed
• pull / linux-focal-py3.8-clang10 / test (dynamo, 1, 2, linux.2xlarge) (gh)
  Process completed with exit code 1.
• pull / linux-focal-py3.8-clang10 / test (dynamo, 2, 2, linux.2xlarge) (gh)
  Process completed with exit code 1.
• pull / linux-jammy-py3.10-clang15-asan / test (default, 1, 6, linux.4xlarge) (gh)
  test_public_bindings.py::TestPublicBindings::test_modules_can_be_imported
• pull / linux-jammy-py3.8-gcc11 / test (default, 1, 3, linux.2xlarge) (gh)
  RuntimeError: inductor/test_debug_trace 1/1 failed
• pull / linux-jammy-py3.8-gcc11 / test (default, 2, 3, linux.2xlarge) (gh)
  RuntimeError: inductor/test_max_autotune 1/1 failed
• pull / linux-jammy-py3.8-gcc11 / test (default, 3, 3, linux.2xlarge) (gh)
  RuntimeError: inductor/test_torchinductor_dynamic_shapes 2/2 failed

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

• inductor / linux-jammy-cpu-py3.8-gcc11-inductor / test (cpu_inductor_huggingface, 1, 1, linux.12xlarge) (gh)
  YituTechConvBert
• inductor / linux-jammy-cpu-py3.8-gcc11-inductor / test (cpu_inductor_timm, 1, 2, linux.12xlarge) (gh)
  mixer_b16_224
• inductor / linux-jammy-cpu-py3.8-gcc11-inductor / test (cpu_inductor_timm, 2, 2, linux.12xlarge) (gh)
  xcit_large_24_p8_224
• inductor / linux-jammy-cpu-py3.8-gcc11-inductor / test (cpu_inductor_torchbench, 2, 2, linux.12xlarge) (gh)
  yolov3
• inductor / linux-jammy-cpu-py3.8-gcc11-inductor / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.12xlarge) (gh)
  YituTechConvBert
• inductor / linux-jammy-cpu-py3.8-gcc11-inductor / test (dynamic_cpu_inductor_timm, 2, 2, linux.12xlarge) (gh)
  xcit_large_24_p8_224
• inductor / linux-jammy-cpu-py3.8-gcc11-inductor / test (dynamic_cpu_inductor_torchbench, 1, 2, linux.12xlarge) (gh)
  hf_T5_base
• inductor / linux-jammy-cpu-py3.8-gcc11-inductor / test (dynamic_cpu_inductor_torchbench, 2, 2, linux.12xlarge) (gh)
  yolov3
• pull / linux-focal-py3_8-clang9-xla / test (xla, 1, 1, linux.12xlarge) (gh)
  ModuleNotFoundError: No module named 'torch.version'
• pull / linux-jammy-py3.8-gcc11 / test (distributed, 1, 2, linux.2xlarge) (gh)
  RuntimeError: distributed/test_inductor_collectives 1/1 failed
• pull / linux-jammy-py3.8-gcc11 / test (distributed, 2, 2, linux.2xlarge) (gh)
  RuntimeError: distributed/_spmd/test_transformation 1/1 failed

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[kadeng]

Moved to a (draft) feature branch, see #115919