[inductor][triton] Update HAS_WARP_SPEC to check triton.Config params. Update Triton Hash to top of release/3.4.x stack #158459

davidberard98 · 2025-07-16T17:40:00Z

Stack from ghstack (oldest at bottom):

-> [inductor][triton] Update HAS_WARP_SPEC to check triton.Config params. Update Triton Hash to top of release/3.4.x stack #158459

Update triton commit hash to 11ec6354315768a85da41032535e3b7b99c5f706, which is the new release/3.4.x branch in triton-lang/triton.

Also, update HAS_WARP_SPEC handling: In triton 3.4, warp spec will have a different interface: num_consumer_groups will be determined automatically by the compiler. This breaks the current Inductor integration, so for now, update HAS_WARP_SPEC to check whether triton.Config takes num_consumer_groups and num_buffers_warp_spec as parameters.

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @chenyang78 @kadeng @muchulee8 @amjames @chauhang @aakhundov

In triton 3.4, warp spec will have a different interface: num_consumer_groups will be determined automatically by the compiler. This breaks the current Inductor integration, so for now, update HAS_WARP_SPEC to check whether triton.Config takes num_consumer_groups and [] as parameters. [ghstack-poisoned]

In triton 3.4, warp spec will have a different interface: num_consumer_groups will be determined automatically by the compiler. This breaks the current Inductor integration, so for now, update HAS_WARP_SPEC to check whether triton.Config takes num_consumer_groups and [] as parameters. ghstack-source-id: 8691928 Pull Request resolved: #158459

pytorch-bot · 2025-07-16T17:40:07Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/158459

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 4 New Failures, 7 Unrelated Failures

As of commit 93b42d6 with merge base e311886 ():

NEW FAILURES - The following jobs have failed:

linux-binary-manywheel / manywheel-py3_9-cuda12_6-test / test (gh)
Process completed with exit code 1.
linux-binary-manywheel / manywheel-py3_9-cuda12_8-test / test (gh)
Process completed with exit code 1.
linux-binary-manywheel / manywheel-py3_9-cuda12_9-test / test (gh)
Process completed with exit code 1.
linux-binary-manywheel-rocm / manywheel-py3_9-rocm6_4-test (gh)
Process completed with exit code 1.

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

inductor / cuda12.8-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) (gh) (trunk failure)
torchrec_dlrm
inductor-rocm / rocm-py3.10-inductor / test (inductor, 1, 2, linux.rocm.gpu.2) (gh) (trunk failure)
distributed/test_dynamo_distributed.py::TestMultiProc::test_ddp_optimizer_cudagraph
inductor-rocm-mi300 / rocm-py3.10-inductor-mi300 / test (inductor, 1, 2, linux.rocm.gpu.mi300.2) (gh) (trunk failure)
distributed/test_dynamo_distributed.py::TestMultiProc::test_ddp_optimizer_cudagraph

UNSTABLE - The following jobs are marked as unstable, possibly due to flakiness on trunk:

pull / cuda12.8-py3.10-gcc9-sm75 / test (pr_time_benchmarks, 1, 1, linux.g4dn.metal.nvidia.gpu, unstable) (gh)
MISSING REGRESSION TEST
rocm / linux-jammy-rocm-py3.10 / test (default, 2, 6, linux.rocm.gpu.2, unstable) (gh) (#156098)
inductor/test_cudagraph_trees.py::CudaGraphTreeTests::test_frozen_fn
rocm / linux-jammy-rocm-py3.10 / test (default, 3, 6, linux.rocm.gpu.2, unstable) (gh) (#156098)
inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_frozen_fn
rocm-mi300 / linux-noble-rocm-py3.12-mi300 / test (default, 3, 6, linux.rocm.gpu.mi300.2, unstable) (gh) (#158182)
inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_frozen_fn

This comment was automatically generated by Dr. CI and updates every 15 minutes.

atalman

lgtm

atalman · 2025-07-16T18:17:47Z

@davidberard98 please set triton hash to: 11ec6354315768a85da41032535e3b7b99c5f706 as in https://github.com/pytorch/pytorch/pull/158442/files

…nfig params" In triton 3.4, warp spec will have a different interface: num_consumer_groups will be determined automatically by the compiler. This breaks the current Inductor integration, so for now, update HAS_WARP_SPEC to check whether triton.Config takes num_consumer_groups and [] as parameters. cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx ipiszy chenyang78 kadeng muchulee8 amjames chauhang aakhundov [ghstack-poisoned]

In triton 3.4, warp spec will have a different interface: num_consumer_groups will be determined automatically by the compiler. This breaks the current Inductor integration, so for now, update HAS_WARP_SPEC to check whether triton.Config takes num_consumer_groups and [] as parameters. ghstack-source-id: df12bd3 Pull Request resolved: #158459

davidberard98 · 2025-07-16T18:20:41Z

@pytorchbot merge

pytorchmergebot · 2025-07-16T18:25:57Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2025-07-16T18:36:53Z

Merge failed

Reason: 6 jobs have failed, first few of them are: docker-builds / docker-build (linux.12xlarge, pytorch-linux-jammy-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks), docker-builds / docker-build (linux.12xlarge, pytorch-linux-jammy-cuda12.8-cudnn9-py3.13-gcc9-inductor-benchmarks), docker-builds / docker-build (linux.12xlarge, pytorch-linux-jammy-cuda12.8-cudnn9-py3-gcc9-inductor-benchmarks), docker-builds / docker-build (linux.12xlarge, pytorch-linux-jammy-cuda12.8-cudnn9-py3.12-gcc9-inductor-benchmarks), docker-builds / docker-build (linux.12xlarge, pytorch-linux-jammy-cuda12.6-cudnn9-py3.12-gcc9-inductor-benchmarks)

Details for Dev Infra team

Raised by workflow job

davidberard98 · 2025-07-16T18:53:33Z

@pytorchbot rebase -b main

pytorchmergebot · 2025-07-16T18:55:01Z

@pytorchbot started a rebase job onto refs/remotes/origin/main. Check the current status here

[ghstack-poisoned]

…nfig params. Update Triton Hash to top of release/3.4.x stack" Update triton commit hash to `11ec6354315768a85da41032535e3b7b99c5f706`, which is the new release/3.4.x branch in triton-lang/triton. Also, update HAS_WARP_SPEC handling: In triton 3.4, warp spec will have a different interface: num_consumer_groups will be determined automatically by the compiler. This breaks the current Inductor integration, so for now, update HAS_WARP_SPEC to check whether triton.Config takes num_consumer_groups and num_buffers_warp_spec as parameters. cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx ipiszy chenyang78 kadeng muchulee8 amjames chauhang aakhundov [ghstack-poisoned]

In triton 3.4, warp spec will have a different interface: num_consumer_groups will be determined automatically by the compiler. This breaks the current Inductor integration, so for now, update HAS_WARP_SPEC to check whether triton.Config takes num_consumer_groups and [] as parameters. ghstack-source-id: d506851 Pull Request resolved: #158459

davidberard98 · 2025-07-17T00:02:11Z

@pytorchbot merge

pytorchmergebot · 2025-07-17T00:04:08Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

…nfig params. Update Triton Hash to top of release/3.4.x stack" Update triton commit hash to `11ec6354315768a85da41032535e3b7b99c5f706`, which is the new release/3.4.x branch in triton-lang/triton. Also, update HAS_WARP_SPEC handling: In triton 3.4, warp spec will have a different interface: num_consumer_groups will be determined automatically by the compiler. This breaks the current Inductor integration, so for now, update HAS_WARP_SPEC to check whether triton.Config takes num_consumer_groups and num_buffers_warp_spec as parameters. cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx ipiszy chenyang78 kadeng muchulee8 amjames chauhang aakhundov [ghstack-poisoned]

In triton 3.4, warp spec will have a different interface: num_consumer_groups will be determined automatically by the compiler. This breaks the current Inductor integration, so for now, update HAS_WARP_SPEC to check whether triton.Config takes num_consumer_groups and [] as parameters. ghstack-source-id: d2f46ea Pull Request resolved: #158459

pytorchmergebot · 2025-07-17T01:02:57Z

Merge failed

Reason: New commits were pushed while merging. Please rerun the merge command.

Details for Dev Infra team

Raised by workflow job

davidberard98 · 2025-07-17T01:08:06Z

@pytorchbot merge

pytorchmergebot · 2025-07-17T01:10:20Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2025-07-17T03:28:23Z

Merge failed

Reason: 1 jobs have failed, first few of them are: linux-binary-manywheel / manywheel-py3_9-cuda12_6-test / test

Details for Dev Infra team

Raised by workflow job

atalman · 2025-07-17T12:48:51Z

@pytorchbot merge -f "lint and workflows are green"

pytorchmergebot · 2025-07-17T12:50:23Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

atalman · 2025-07-17T18:58:01Z

@pytorchbot cherry-pick --onto release/2.8 -c release

pytorchbot · 2025-07-17T19:02:42Z

Cherry picking #158459

Command git -C /home/runner/work/pytorch/pytorch cherry-pick -x 7892f5a007859ae02b9cac5441cd51f37147ef04 returned non-zero exit code 1

Auto-merging test/inductor/test_static_cuda_launcher.py
CONFLICT (content): Merge conflict in test/inductor/test_static_cuda_launcher.py
Auto-merging torch/_inductor/runtime/triton_compat.py
error: could not apply 7892f5a0078... [inductor][triton] Update HAS_WARP_SPEC to check triton.Config params. Update Triton Hash to top of release/3.4.x stack (#158459)
hint: After resolving the conflicts, mark them with
hint: "git add/rm <pathspec>", then run
hint: "git cherry-pick --continue".
hint: You can instead skip this commit with "git cherry-pick --skip".
hint: To abort and get back to the state before "git cherry-pick",
hint: run "git cherry-pick --abort".
hint: Disable this message with "git config set advice.mergeConflict false"

Details for Dev Infra team

Raised by workflow job

…. Update Triton Hash to top of release/3.4.x stack (pytorch#158459) Update triton commit hash to `11ec6354315768a85da41032535e3b7b99c5f706`, which is the new release/3.4.x branch in triton-lang/triton. Also, update HAS_WARP_SPEC handling: In triton 3.4, warp spec will have a different interface: num_consumer_groups will be determined automatically by the compiler. This breaks the current Inductor integration, so for now, update HAS_WARP_SPEC to check whether triton.Config takes num_consumer_groups and num_buffers_warp_spec as parameters. Pull Request resolved: pytorch#158459 Approved by: https://github.com/atalman

…Config params. Update Triton Hash to top of release/3.4.x stack (#158646) * [inductor][triton] Update HAS_WARP_SPEC to check triton.Config params. Update Triton Hash to top of release/3.4.x stack (#158459) Update triton commit hash to `11ec6354315768a85da41032535e3b7b99c5f706`, which is the new release/3.4.x branch in triton-lang/triton. Also, update HAS_WARP_SPEC handling: In triton 3.4, warp spec will have a different interface: num_consumer_groups will be determined automatically by the compiler. This breaks the current Inductor integration, so for now, update HAS_WARP_SPEC to check whether triton.Config takes num_consumer_groups and num_buffers_warp_spec as parameters. Pull Request resolved: #158459 Approved by: https://github.com/atalman * dont_upde_hash * Revert "dont_upde_hash" This reverts commit 5fffb12. * fix_docker_builds --------- Co-authored-by: David Berard <dberard@fb.com>

pytorch-bot bot added ciflow/inductor module: inductor labels Jul 16, 2025

davidberard98 added the release notes: inductor label Jul 16, 2025

atalman approved these changes Jul 16, 2025

View reviewed changes

davidberard98 requested review from desertfire, Chillee, eellison, shunting314, bertmaher, jeffdaily, jataylo, jithunnair-amd and pruthvistony as code owners July 16, 2025 18:19

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Jul 16, 2025

davidberard98 changed the title ~~[inductor][triton] Update HAS_WARP_SPEC to check triton.Config params~~ [triton] New 3.4 release pin && update HAS_WARP_SPEC to check triton.Config params Jul 16, 2025

atalman changed the title ~~[triton] New 3.4 release pin && update HAS_WARP_SPEC to check triton.Config params~~ [inductor][triton] Update HAS_WARP_SPEC to check triton.Config params. Update Triton Hash to top of release/3.4.x stack Jul 16, 2025

pytorchmergebot added the merging label Jul 16, 2025

pytorchmergebot removed the merging label Jul 16, 2025

Update

9524e46

[ghstack-poisoned]

atalman added the ci-no-td Do not run TD on this PR label Jul 16, 2025

pytorchmergebot added the merging label Jul 17, 2025

pytorchmergebot removed the merging label Jul 17, 2025

davidberard98 mentioned this pull request Jul 17, 2025

[static cuda launcher] scalar float support #158521

Open

pytorchmergebot added the merging label Jul 17, 2025

pytorchmergebot removed the merging label Jul 17, 2025

davidberard98 added the ciflow/triton_binaries label Jul 17, 2025

pytorchmergebot added the merging label Jul 17, 2025

pytorchmergebot closed this in 7892f5a Jul 17, 2025

pytorchmergebot added Merged and removed merging labels Jul 17, 2025

This was referenced Jul 18, 2025

[cherry-pick][inductor][triton] Update HAS_WARP_SPEC to check triton.Config params. Update Triton Hash to top of release/3.4.x stack #158646

Merged

[v.2.8.0] Release Tracker #156745

Open

[inductor][triton] Update HAS_WARP_SPEC to check triton.Config params. Update Triton Hash to top of release/3.4.x stack #158459

[inductor][triton] Update HAS_WARP_SPEC to check triton.Config params. Update Triton Hash to top of release/3.4.x stack #158459

Uh oh!

Conversation

davidberard98 commented Jul 16, 2025 • edited by pytorchmergebot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Jul 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/158459

❌ 4 New Failures, 7 Unrelated Failures

Uh oh!

atalman left a comment

Choose a reason for hiding this comment

Uh oh!

atalman commented Jul 16, 2025

Uh oh!

davidberard98 commented Jul 16, 2025

Uh oh!

pytorchmergebot commented Jul 16, 2025

Merge started

Uh oh!

pytorchmergebot commented Jul 16, 2025

Merge failed

Uh oh!

davidberard98 commented Jul 16, 2025

Uh oh!

pytorchmergebot commented Jul 16, 2025

Uh oh!

davidberard98 commented Jul 17, 2025

Uh oh!

pytorchmergebot commented Jul 17, 2025

Merge started

Uh oh!

pytorchmergebot commented Jul 17, 2025

Merge failed

Uh oh!

davidberard98 commented Jul 17, 2025

Uh oh!

pytorchmergebot commented Jul 17, 2025

Merge started

Uh oh!

pytorchmergebot commented Jul 17, 2025

Merge failed

Uh oh!

atalman commented Jul 17, 2025

Uh oh!

pytorchmergebot commented Jul 17, 2025

Merge started

Uh oh!

atalman commented Jul 17, 2025

Uh oh!

pytorchbot commented Jul 17, 2025

Cherry picking #158459

Uh oh!

Uh oh!

davidberard98 commented Jul 16, 2025 •

edited by pytorchmergebot

Loading

pytorch-bot bot commented Jul 16, 2025 •

edited

Loading