[inductor max autotune] Support sigmoid and tanh activations

[kadeng]

Stack from ghstack (oldest at bottom):

• [Inductor cutlass backend] Fixed workspace resize issue #115877
• [Inductor cutlass backend] Log CUDA compilation times #115814
• [Inductor cutlass backend] Added (some) doc comments and type annotations #115813
• [Inductor cutlass backend] Test and fix broadcasting of Aux / Bias edge cases #115655
• [Inductor cutlass backend] Robust Precompilation / Autotuning / Retuning in subprocesses #115654
• [Cutlass 3.3 submodule upgrade] #112861
• [Inductor cutlass backend] Support for StreamK and non-zero workspace sizes #114687
• [Inductor cutlass backend] Add support for row/column broadcasting of auxiliary inputs #115270
• [Inductor cutlass backend] Provide CUDA SM count to kernels #115174
• [Inductor cutlass backend] Generate standalone debug runners for Cutlass GEMMs #115072
• -> [inductor max autotune] Support sigmoid and tanh activations #114606
• [inductor max autotune] Flexible GEMM layout autotuning #114319
• [Inductor cutlass backend] Allow to fall back to non-fuseable ops #114075
• [Inductor cutlass backend] Experimental extended Cutlass op generator #113932
• [inductor max autotune] Log but continue on errors during autotuning benchmark #113891
• [Inductor Cutlass backend] Enabling flexible EVT-based pointwise fusions with additional tensor input #113959
• [Inductor cutlass backend] Enable torch.bmm and torch.baddbmm support #113890
• [Inductor cutlass backend] disable StreamK for GEMMs due to performance issues #113570
• [Cutlass inductor backend] Cutlass GEMM size threshold #113569
• [Inductor max autotune] Multithreaded Precompilation #113558
• [inductor max autotune] Detailed autotuning result logs ( machine-readable ) #113399
• [Inductor cutlass GEMM backend] Fix for operand memory layout change between autotuning and CUTLASSGEMMTemplate.render #113366
• [Inductor] Fix debug_str method of FusedSchedulerNode #113365

Simple change adding support for Sigmoid and Tanh activations.

Test Plan:

• CI
• Additional unit test ( tbd )

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/114606

• 📄 Preview Python docs built from this PR
• 📄 Preview C++ docs built from this PR
• ❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 22 New Failures, 10 Unrelated Failures

As of commit 7cefb0d with merge base afe6d27 ():

NEW FAILURES - The following jobs have failed:

• inductor / linux-jammy-cpu-py3.8-gcc11-inductor / test (cpu_inductor_torchbench, 1, 2, linux.12xlarge) (gh)
  hf_T5_base
• Lint / lintrunner / linux-job (gh)
  >>> Lint for torch/_inductor/codegen/cuda/gemm_template.py:
• pull / linux-docs / build-docs-python-false (gh)
  Process completed with exit code 2.
• pull / linux-focal-cuda12.1-py3.10-gcc9 / test (default, 1, 5, linux.4xlarge.nvidia.gpu) (gh)
  test_public_bindings.py::TestPublicBindings::test_modules_can_be_imported
• pull / linux-focal-cuda12.1-py3.10-gcc9-sm86 / test (default, 1, 5, linux.g5.4xlarge.nvidia.gpu) (gh)
  inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_bmm_multiple_dynamic_abi_compatible_cpu
• pull / linux-focal-cuda12.1-py3.10-gcc9-sm86 / test (default, 2, 5, linux.g5.4xlarge.nvidia.gpu) (gh)
  inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpu::test_addmm_multiple_dynamic_abi_compatible_cpu
• pull / linux-focal-cuda12.1-py3.10-gcc9-sm86 / test (default, 5, 5, linux.g5.4xlarge.nvidia.gpu) (gh)
  inductor/test_aot_inductor.py::AOTInductorTestABICompatibleCpuWithStackAllocationAndMinimalArrayRefInterface::test_addmm_multiple_dynamic_abi_compatible_cpu_with_stack_allocation_and_minimal_arrayref_interface
• pull / linux-focal-py3.11-clang10 / test (crossref, 1, 2, linux.2xlarge) (gh)
  test_public_bindings.py::TestPublicBindings::test_modules_can_be_imported
• pull / linux-focal-py3.11-clang10 / test (default, 1, 3, linux.2xlarge) (gh)
  test_public_bindings.py::TestPublicBindings::test_modules_can_be_imported
• pull / linux-focal-py3.8-clang10 / test (crossref, 1, 2, linux.2xlarge) (gh)
  RuntimeError: inductor/test_debug_trace 1/1 failed
• pull / linux-focal-py3.8-clang10 / test (crossref, 2, 2, linux.2xlarge) (gh)
  test_testing.py::TestImports::test_circular_dependencies
• pull / linux-focal-py3.8-clang10 / test (default, 1, 3, linux.2xlarge) (gh)
  RuntimeError: inductor/test_debug_trace 1/1 failed
• pull / linux-focal-py3.8-clang10 / test (default, 2, 3, linux.2xlarge) (gh)
  RuntimeError: inductor/test_max_autotune 1/1 failed
• pull / linux-focal-py3.8-clang10 / test (default, 3, 3, linux.2xlarge) (gh)
  RuntimeError: inductor/test_torchinductor_dynamic_shapes 1/2 failed
• pull / linux-focal-py3.8-clang10 / test (dynamo, 1, 2, linux.2xlarge) (gh)
  Process completed with exit code 1.
• pull / linux-focal-py3.8-clang10 / test (dynamo, 2, 2, linux.2xlarge) (gh)
  Process completed with exit code 1.
• pull / linux-jammy-py3.10-clang15-asan / test (default, 1, 6, linux.4xlarge) (gh)
  test_public_bindings.py::TestPublicBindings::test_modules_can_be_imported
• pull / linux-jammy-py3.8-gcc11 / test (default, 1, 3, linux.2xlarge) (gh)
  RuntimeError: inductor/test_debug_trace 1/1 failed
• pull / linux-jammy-py3.8-gcc11 / test (default, 2, 3, linux.2xlarge) (gh)
  RuntimeError: inductor/test_max_autotune 1/1 failed
• pull / linux-jammy-py3.8-gcc11 / test (default, 3, 3, linux.2xlarge) (gh)
  RuntimeError: inductor/test_torchinductor_dynamic_shapes 1/2 failed
• pull / linux-jammy-py3.8-gcc11 / test (distributed, 1, 2, linux.2xlarge) (gh)
  RuntimeError: distributed/test_inductor_collectives 1/1 failed
• pull / linux-jammy-py3.8-gcc11 / test (distributed, 2, 2, linux.2xlarge) (gh)
  RuntimeError: distributed/_spmd/test_transformation 1/1 failed

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

• inductor / linux-jammy-cpu-py3.8-gcc11-inductor / test (cpu_inductor_huggingface, 1, 1, linux.12xlarge) (gh)
  YituTechConvBert
• inductor / linux-jammy-cpu-py3.8-gcc11-inductor / test (cpu_inductor_timm, 1, 2, linux.12xlarge) (gh)
  mixer_b16_224
• inductor / linux-jammy-cpu-py3.8-gcc11-inductor / test (cpu_inductor_timm, 2, 2, linux.12xlarge) (gh)
  xcit_large_24_p8_224
• inductor / linux-jammy-cpu-py3.8-gcc11-inductor / test (cpu_inductor_torchbench, 2, 2, linux.12xlarge) (gh)
  yolov3
• inductor / linux-jammy-cpu-py3.8-gcc11-inductor / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.12xlarge) (gh)
  YituTechConvBert
• inductor / linux-jammy-cpu-py3.8-gcc11-inductor / test (dynamic_cpu_inductor_timm, 1, 2, linux.12xlarge) (gh)
  mixer_b16_224
• inductor / linux-jammy-cpu-py3.8-gcc11-inductor / test (dynamic_cpu_inductor_timm, 2, 2, linux.12xlarge) (gh)
  xcit_large_24_p8_224
• inductor / linux-jammy-cpu-py3.8-gcc11-inductor / test (dynamic_cpu_inductor_torchbench, 1, 2, linux.12xlarge) (gh)
  hf_T5_base
• inductor / linux-jammy-cpu-py3.8-gcc11-inductor / test (dynamic_cpu_inductor_torchbench, 2, 2, linux.12xlarge) (gh)
  yolov3
• pull / linux-focal-py3_8-clang9-xla / test (xla, 1, 1, linux.12xlarge) (gh)
  ModuleNotFoundError: No module named 'torch.version'

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[kadeng]

Moved to a (draft) feature branch, see #115919