[Inductor cutlass backend] Robust Precompilation / Autotuning / Retuning in subprocesses

[kadeng]

Stack from ghstack (oldest at bottom):

• [Inductor cutlass backend] Fixed workspace resize issue #115877
• [Inductor cutlass backend] Log CUDA compilation times #115814
• [Inductor cutlass backend] Added (some) doc comments and type annotations #115813
• [Inductor cutlass backend] Test and fix broadcasting of Aux / Bias edge cases #115655
• -> [Inductor cutlass backend] Robust Precompilation / Autotuning / Retuning in subprocesses #115654
• [Cutlass 3.3 submodule upgrade] #112861
• [Inductor cutlass backend] Support for StreamK and non-zero workspace sizes #114687
• [Inductor cutlass backend] Add support for row/column broadcasting of auxiliary inputs #115270
• [Inductor cutlass backend] Provide CUDA SM count to kernels #115174
• [Inductor cutlass backend] Generate standalone debug runners for Cutlass GEMMs #115072
• [inductor max autotune] Support sigmoid and tanh activations #114606
• [inductor max autotune] Flexible GEMM layout autotuning #114319
• [Inductor cutlass backend] Allow to fall back to non-fuseable ops #114075
• [Inductor cutlass backend] Experimental extended Cutlass op generator #113932
• [inductor max autotune] Log but continue on errors during autotuning benchmark #113891
• [Inductor Cutlass backend] Enabling flexible EVT-based pointwise fusions with additional tensor input #113959
• [Inductor cutlass backend] Enable torch.bmm and torch.baddbmm support #113890
• [Inductor cutlass backend] disable StreamK for GEMMs due to performance issues #113570
• [Cutlass inductor backend] Cutlass GEMM size threshold #113569
• [Inductor max autotune] Multithreaded Precompilation #113558
• [inductor max autotune] Detailed autotuning result logs ( machine-readable ) #113399
• [Inductor cutlass GEMM backend] Fix for operand memory layout change between autotuning and CUTLASSGEMMTemplate.render #113366
• [Inductor] Fix debug_str method of FusedSchedulerNode #113365

Makes autotuning in subprocesses more robust, specifically against long running or crashing
functions being benchmarked, which could also completely corrupt the CUDA Context of the entire process.

This diff introduces changes to ensure that precompilation works well with autotuning in
subprocesses, and ensures that autotuning subprocesses have robust timeouts after which
they will be killed.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/115654

• 📄 Preview Python docs built from this PR
• 📄 Preview C++ docs built from this PR
• ❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 20 New Failures, 12 Unrelated Failures

As of commit f7c2cf2 with merge base afe6d27 ():

NEW FAILURES - The following jobs have failed:

• Lint / lintrunner / linux-job (gh)
  >>> Lint for torch/_inductor/scheduler.py:
• pull / linux-docs / build-docs-python-false (gh)
  Process completed with exit code 2.
• pull / linux-focal-cuda12.1-py3.10-gcc9 / test (default, 1, 5, linux.4xlarge.nvidia.gpu) (gh)
  test_public_bindings.py::TestPublicBindings::test_modules_can_be_imported
• pull / linux-focal-cuda12.1-py3.10-gcc9-sm86 / test (default, 1, 5, linux.g5.4xlarge.nvidia.gpu) (gh)
  inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_bmm_multiple_dynamic_abi_compatible_cpu
• pull / linux-focal-cuda12.1-py3.10-gcc9-sm86 / test (default, 2, 5, linux.g5.4xlarge.nvidia.gpu) (gh)
  inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_addmm_multiple_dynamic_abi_compatible_cpu
• pull / linux-focal-cuda12.1-py3.10-gcc9-sm86 / test (default, 5, 5, linux.g5.4xlarge.nvidia.gpu) (gh)
  inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpuWithStackAllocationAndMinimalArrayRefInterface::test_addmm_multiple_dynamic_abi_compatible_cpu_with_stack_allocation_and_minimal_arrayref_interface
• pull / linux-focal-py3.11-clang10 / test (crossref, 1, 2, linux.2xlarge) (gh)
  test_public_bindings.py::TestPublicBindings::test_modules_can_be_imported
• pull / linux-focal-py3.11-clang10 / test (default, 1, 3, linux.2xlarge) (gh)
  test_public_bindings.py::TestPublicBindings::test_modules_can_be_imported
• pull / linux-focal-py3.8-clang10 / test (crossref, 1, 2, linux.2xlarge) (gh)
  RuntimeError: inductor/test_debug_trace 1/1 failed
• pull / linux-focal-py3.8-clang10 / test (crossref, 2, 2, linux.2xlarge) (gh)
  RuntimeError: inductor/test_torchinductor 1/1 failed
• pull / linux-focal-py3.8-clang10 / test (default, 1, 3, linux.2xlarge) (gh)
  RuntimeError: inductor/test_debug_trace 1/1 failed
• pull / linux-focal-py3.8-clang10 / test (default, 2, 3, linux.2xlarge) (gh)
  RuntimeError: inductor/test_max_autotune 1/1 failed
• pull / linux-focal-py3.8-clang10 / test (default, 3, 3, linux.2xlarge) (gh)
  RuntimeError: inductor/test_torchinductor_dynamic_shapes 2/2 failed
• pull / linux-focal-py3.8-clang10 / test (dynamo, 1, 2, linux.2xlarge) (gh)
  Process completed with exit code 1.
• pull / linux-jammy-py3.10-clang15-asan / test (default, 1, 6, linux.4xlarge) (gh)
  test_public_bindings.py::TestPublicBindings::test_modules_can_be_imported
• pull / linux-jammy-py3.8-gcc11 / test (default, 1, 3, linux.2xlarge) (gh)
  RuntimeError: inductor/test_debug_trace 1/1 failed
• pull / linux-jammy-py3.8-gcc11 / test (default, 2, 3, linux.2xlarge) (gh)
  RuntimeError: inductor/test_max_autotune 1/1 failed
• pull / linux-jammy-py3.8-gcc11 / test (default, 3, 3, linux.2xlarge) (gh)
  RuntimeError: inductor/test_torchinductor_dynamic_shapes 2/2 failed
• pull / linux-jammy-py3.8-gcc11 / test (distributed, 1, 2, linux.2xlarge) (gh)
  RuntimeError: distributed/test_inductor_collectives 1/1 failed
• pull / linux-jammy-py3.8-gcc11 / test (distributed, 2, 2, linux.2xlarge) (gh)
  RuntimeError: distributed/_spmd/test_transformation 1/1 failed

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

• inductor / linux-jammy-cpu-py3.8-gcc11-inductor / test (cpu_inductor_huggingface, 1, 1, linux.12xlarge) (gh)
  YituTechConvBert
• inductor / linux-jammy-cpu-py3.8-gcc11-inductor / test (cpu_inductor_timm, 1, 2, linux.12xlarge) (gh)
  mixer_b16_224
• inductor / linux-jammy-cpu-py3.8-gcc11-inductor / test (cpu_inductor_timm, 2, 2, linux.12xlarge) (gh)
  xcit_large_24_p8_224
• inductor / linux-jammy-cpu-py3.8-gcc11-inductor / test (cpu_inductor_torchbench, 1, 2, linux.12xlarge) (gh)
  hf_T5_base
• inductor / linux-jammy-cpu-py3.8-gcc11-inductor / test (cpu_inductor_torchbench, 2, 2, linux.12xlarge) (gh)
  yolov3
• inductor / linux-jammy-cpu-py3.8-gcc11-inductor / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.12xlarge) (gh)
  YituTechConvBert
• inductor / linux-jammy-cpu-py3.8-gcc11-inductor / test (dynamic_cpu_inductor_timm, 1, 2, linux.12xlarge) (gh)
  mixer_b16_224
• inductor / linux-jammy-cpu-py3.8-gcc11-inductor / test (dynamic_cpu_inductor_timm, 2, 2, linux.12xlarge) (gh)
  xcit_large_24_p8_224
• inductor / linux-jammy-cpu-py3.8-gcc11-inductor / test (dynamic_cpu_inductor_torchbench, 1, 2, linux.12xlarge) (gh)
  hf_T5_base
• inductor / linux-jammy-cpu-py3.8-gcc11-inductor / test (dynamic_cpu_inductor_torchbench, 2, 2, linux.12xlarge) (gh)
  yolov3
• pull / linux-focal-py3_8-clang9-xla / test (xla, 1, 1, linux.12xlarge) (gh)
  ModuleNotFoundError: No module named 'torch.version'
• pull / linux-focal-py3.8-clang10 / test (dynamo, 2, 2, linux.2xlarge) (gh)
  Process completed with exit code 1.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[kadeng]

Moved to a (draft) feature branch, see #115919