[inductor] Support atomic template reduction groups

[mlazos]

Stack from ghstack (oldest at bottom):

• [inductor][NVGEMM] Use direct CuTeDSL lowering for dense epilogues #191515
• [inductor][NVGEMM] Adapt dense EFC to direct CuTeDSL epilogues #191514
• [inductor][NVGEMM] Add grouped epilogue parity coverage #190825
• -> [inductor] Support atomic template reduction groups #190822
• [inductor][NVGEMM] Register vendored epilogue providers #190821
• [inductor][NVGEMM] Extend block-scaled kernel reduction mechanics #190820
• [inductor][NVGEMM] Vendor the SM100 dense EFC kernel #190819

Allow template scheduling to claim a complete reduction chain as one atomic fusion unit and retain the template output identity needed by multi-output kernels. Prevent partial fusion from consuming layout or finalizer nodes before NVGEMM can validate the complete grouped pattern.

Reject horizontal fusion involving NVGEMM boolean outputs and prioritize valid NVGEMM reduction chains ahead of ordinary Triton fusion candidates.

Test Plan:

lintrunner -a

Authored with an AI assistant.

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @kadeng @muchulee8 @amjames @chauhang @aakhundov @coconutruben @jataylo

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/190822

• 📄 Preview Python docs built from this PR
• 📄 Preview C++ docs built from this PR
• ❓ Need help or want to give feedback on the CI? Visit the bot commands wiki

Note: Links to docs will display an error until the docs builds have been completed.

❌ 9 New Failures, 4 Cancelled Jobs

As of commit 8c17d64 with merge base e56e791 ():

NEW FAILURES - The following jobs have failed:

• inductor / unit-test / inductor-cpu-core-test (3.13) / test (inductor_core, 1, 2, mt-l-x86iavx512-8-64) (gh)
  test/inductor/test_nv_universal_gemm.py::TestNVUniversalGemmHeuristics::test_epilogue_program_owns_only_named_feed_intermediates
• inductor / unit-test / inductor-cpu-core-test (3.13) / test (inductor_core, 2, 2, mt-l-x86iavx512-8-64) (gh)
  test/inductor/test_nv_universal_gemm.py::TestNVUniversalGemmHeuristics::test_epilogue_program_owns_only_named_feed_intermediates
• Lint / pr-sanity-checks (gh)
  LOC which is more than the 2000 maximum
• pull / linux-jammy-aarch64-py3.10 / test (default, 2, 7, mt-l-arm64g4-16-62) (gh)
  test/inductor/test_nv_universal_gemm.py::TestNVUniversalGemmHeuristics::test_epilogue_program_owns_only_named_feed_intermediates
• pull / linux-jammy-py3.10-clang21 / test (default, 4, 10, mt-l-x86iavx512-16-128) (gh)
  test/inductor/test_nv_universal_gemm.py::TestNVUniversalGemmHeuristics::test_epilogue_program_owns_only_named_feed_intermediates
• pull / linux-jammy-py3.10-clang21-asan / test (default, 5, 18, mt-l-x86iavx512-16-128) (gh)
  test/inductor/test_nv_universal_gemm.py::TestNVUniversalGemmHeuristics::test_epilogue_program_owns_only_named_feed_intermediates
• pull / linux-jammy-py3.10-gcc11 / test (default, 4, 10, mt-l-x86iavx512-16-128) (gh)
  test/inductor/test_nv_universal_gemm.py::TestNVUniversalGemmHeuristics::test_epilogue_program_owns_only_named_feed_intermediates
• pull / linux-jammy-py3.14-clang21 / test (default, 4, 10, mt-l-x86iavx512-16-128) (gh)
  test/inductor/test_nv_universal_gemm.py::TestNVUniversalGemmHeuristics::test_epilogue_program_owns_only_named_feed_intermediates
• pull / linux-jammy-py3.14t-clang21 / test (default, 4, 11, mt-l-x86iavx512-16-128) (gh)
  test/inductor/test_nv_universal_gemm.py::TestNVUniversalGemmHeuristics::test_epilogue_program_owns_only_named_feed_intermediates

CANCELLED JOBS - The following jobs were cancelled. Please retry:

• inductor / unit-test / inductor-cpu-core-test (3.11) / test (inductor_core, 1, 2, mt-l-x86iavx512-8-64) (gh)
  [runner-container-hooks] FATAL: received SIGINT
• inductor / unit-test / inductor-cpu-core-test (3.11) / test (inductor_core, 2, 2, mt-l-x86iavx512-8-64) (gh)
  API rate limit exceeded
• inductor / unit-test / inductor-cpu-core-test (3.12) / test (inductor_core, 1, 2, mt-l-x86iavx512-8-64) (gh)
  API rate limit exceeded
• inductor / unit-test / inductor-cpu-core-test (3.12) / test (inductor_core, 2, 2, mt-l-x86iavx512-8-64) (gh)
  ##[error]received SIGINT

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[mlazos]

AI-Review

Automated multi-agent review (2 Claude + 2 Codex + synthesis). This is AI-generated — please verify findings manually.

---

Verified Issues

• [medium] Fusion can commit without a tile-compatible EFC choice — Location: can_fuse_reduction_epilogue(), codegen_template(), _best_nvgemm_choice(). Trigger: an axis-1 grouped reduction whose group exceeds every available EFC tile-N. Failure sequence: fusion returns True; codegen filters out every choice using min_tile_n; _best_nvgemm_choice() raises RuntimeError. Impact: compilation aborts instead of falling back. Reviewers: Claude Design, Codex Design, Codex Bug.

• [low] New test accesses a nonexistent property — Location: test_epilogue_program_owns_only_named_feed_intermediates. Trigger: run the test. Failure sequence: it evaluates program.pointwise_nodes, but NVGemmEpilogueProgram only defines evt_nodes; attribute lookup raises AttributeError. Impact: deterministic test failure. Reviewers: none; found during verification.