[torch][windows] Build Windows release PyTorch wheels. #994

ScottTodd · 2025-07-09T23:29:33Z

Progress on #827 (!!)

Tested at https://github.com/ROCm/TheRock/actions/runs/16185309410, which triggered these:

https://github.com/ROCm/TheRock/actions/runs/16186383600 (gfx1151)
https://github.com/ROCm/TheRock/actions/runs/16186941663 (gfx120X-all)
https://github.com/ROCm/TheRock/actions/runs/16186966931 (gfx110X-dgpu)

Some next steps after this, roughly in order:

Monitor scheduled jobs to check that wheels are successfully built and uploaded
Download/install and test the wheels on various systems, exercising components like hipBLASLt
Update support status:
- https://github.com/ROCm/TheRock?tab=readme-ov-file#support-status
- https://github.com/ROCm/TheRock/blob/main/RELEASES.md#python-packages-support-status
Build torchvision and torchaudio (need some patches)
Connect with other build/test/release workflows so we get pre-submit, post-submit, and scheduled coverage for these builds/releases/tests
Rebase patches onto PyTorch 2.8 (and/or HEAD), land patches upstream
Add features like Triton support (aotriton?) for optimized flash attention kernels, etc.
Expand the list of supported gfx targets

This depends on some other open PRs, but I have tested parts of it all the way through to dev wheel publishing and then installation and testing on a dev machine: https://github.com/ROCm/TheRock/actions/runs/16181411431/job/45678668536, #827 (comment).

~~See the other PRs:~~

marbre · 2025-07-09T23:35:34Z

.github/workflows/release_windows_pytorch_wheels.yml

  # TODO(scotttodd): add schedule once working
  # schedule:


FYI, I dropped running on schedule from the Linux workflow and instead trigger this workflow from the release portable Linux packages workflow.

Good catch, I'll make those updates. I had been testing exclusively with workflow_dispatch.

Copied over changes from #983 to these files. I'll want to test before marking this ready for review/merge... once the dependent PRs are merged.

marbre · 2025-07-09T23:36:18Z

.github/workflows/release_windows_pytorch_wheels.yml

+      release_type: ${{ inputs.release_type || 'nightly' }}
+      s3_subdir: ${{ inputs.s3_subdir || 'v2' }}
+      cloudfront_url: ${{ inputs.cloudfront_url  || 'd2awnip2yjpvqn.cloudfront.net/v2' }}


The defaults can be dropped if not planning to run on schedule.

stellaraccident

build_prod_wheels changes look fine to me. Pipeline changes as well, but I am not the best reviewer on those.

…dows-nightly-pytorch-5

ScottTodd · 2025-07-10T00:11:54Z

Dependent PRs are all merged now. I'll trigger a dev release of the full pipeline, work through any remaining workflow issues, then mark ready for review hopefully tomorrow.

edit: https://github.com/ROCm/TheRock/actions/runs/16182932309 is queued up now

Previously I had those in a standalone step.

ScottTodd · 2025-07-10T14:38:49Z

Ready for review now!

The test dev packages build at https://github.com/ROCm/TheRock/actions/runs/16185309410 succeeded then triggered these follow-on workflows runs, which also succeeded:

https://github.com/ROCm/TheRock/actions/runs/16186383600 (gfx1151)
https://github.com/ROCm/TheRock/actions/runs/16186941663 (gfx120X-all)
https://github.com/ROCm/TheRock/actions/runs/16186966931 (gfx110X-dgpu)

I now see wheels on the dev index pages like https://d25kgig7rdsyks.cloudfront.net/v2/gfx110X-dgpu/torch/ for python 3.11, 3.12, and 3.13. I'll sanity check those shortly.

ScottTodd · 2025-07-10T15:17:25Z

Sanity check on gfx1100 passed, except for this known issue where the smoketests do not terminate after passing: #999

marbre

LGTM, nice to see this landing!

marbre · 2025-07-10T15:41:36Z

.github/workflows/build_windows_pytorch_wheels.yml

+        shell: cmd
+        run: |
+          echo "Building PyTorch wheels for ${{ inputs.amdgpu_family }}"
+          python ./external-builds/pytorch/build_prod_wheels.py build --install-rocm --index-url "https://${{ inputs.cloudfront_url }}/${{ inputs.amdgpu_family }}/" --pytorch-dir ${{ env.CHECKOUT_ROOT }}/torch --clean --output-dir ${{ env.PACKAGE_DIST_DIR }} ${{ env.optional_build_prod_arguments }}


Suggested change

python ./external-builds/pytorch/build_prod_wheels.py build --install-rocm --index-url "https://${{ inputs.cloudfront_url }}/${{ inputs.amdgpu_family }}/" --pytorch-dir ${{ env.CHECKOUT_ROOT }}/torch --clean --output-dir ${{ env.PACKAGE_DIST_DIR }} ${{ env.optional_build_prod_arguments }}

python ./external-builds/pytorch/build_prod_wheels.py \

build \

--install-rocm \

--index-url "https://${{ inputs.cloudfront_url }}/${{ inputs.amdgpu_family }}/" \

--pytorch-dir ${{ env.CHECKOUT_ROOT }}/torch \

--clean \

--output-dir ${{ env.PACKAGE_DIST_DIR }} ${{ env.optional_build_prod_arguments }}

This would follow the style in .github/workflows/build_linux_pytorch_wheels.yml and might be easier to read.

I wish... the single line is intentional given that the shell is cmd and not bash.

I think I can use ^ instead though...

https://stackoverflow.com/questions/69068/split-long-commands-in-multiple-lines-through-windows-batch-file

Oh, above you're setting shell: bash for the entire job or not?

I'm switching between cmd and bash throughout the file 🙈. When I build locally I use cmd exclusively but I had some trouble with each shell for different steps on the CI runners. I'd like to go through and choose one consistently in the workflow... after checkpointing and getting the release train rolling.

Yeah, I see... you could have dropped shell: cmd here to make it work but I see that this makes copying over and testing locally unnecessary more difficult. Whatever works best for you :)

Using bash here would allow the \ characters to split the command across multiple lines, but the build process fails under bash (on my local machine and CI: #827 (comment)). That should be fixable, but it's a battle for another day.

It might be easier to get the rest of the workflow using cmd.

I'll try with ^ and add some breadcrumb comments where the shell is load bearing

Seems to be working with ^: https://github.com/ROCm/TheRock/actions/runs/16201014899/job/45739794416

marbre · 2025-07-10T15:45:09Z

.github/workflows/release_windows_pytorch_wheels.yml

-        python_version: ["3.11", "3.12"]
+        python_version: ["3.11", "3.12", "3.13"]

    uses: ./.github/workflows/build_windows_pytorch_wheels.yml


I recently revised this workflow further (PR #998) and expect for this particular line the workflows are identically. Could merge them and rather add an input parameter that tells if to build for Linux or Windows maybe?.

Yeah, merging should be possible now :D

Well we can go for it as is, this isn't blocking but wanted to flag that we can clean in a follow up :)

Nem404 · 2025-07-10T17:33:48Z

Yes, aotriton support would be a good idea too, as people say who use scottt and jammm's wheels, it gives a considerable performance boost

ScottTodd added 2 commits July 9, 2025 16:17

[torch][windows] Build Windows release PyTorch wheels.

09d3c1e

Cleanup workflow after debugging.

e23bab3

github-project-automation bot added this to TheRock Triage Jul 9, 2025

github-project-automation bot moved this to TODO in TheRock Triage Jul 9, 2025

ScottTodd requested review from marbre and stellaraccident July 9, 2025 23:29

marbre reviewed Jul 9, 2025

View reviewed changes

Adapt triggering changes from Linux workflows.

71c34ec

stellaraccident approved these changes Jul 10, 2025

View reviewed changes

Merge remote-tracking branch 'upstream/main' into users/scotttodd/win…

aa2619b

…dows-nightly-pytorch-5

Add --install-rocm and --index-url to build_prod_wheels command.

e96d36b

Previously I had those in a standalone step.

ScottTodd marked this pull request as ready for review July 10, 2025 14:38

ScottTodd requested a review from marbre July 10, 2025 14:38

marbre approved these changes Jul 10, 2025

View reviewed changes

Use ^ to wrap lines when using cmd, document other uses.

85934ad

ScottTodd merged commit 11d7c75 into main Jul 10, 2025
4 of 5 checks passed

github-project-automation bot moved this from TODO to Done in TheRock Triage Jul 10, 2025

ScottTodd deleted the users/scotttodd/windows-nightly-pytorch-5 branch July 10, 2025 17:10

-          python ./external-builds/pytorch/build_prod_wheels.py build --install-rocm --index-url "https://${{ inputs.cloudfront_url }}/${{ inputs.amdgpu_family }}/" --pytorch-dir ${{ env.CHECKOUT_ROOT }}/torch --clean --output-dir ${{ env.PACKAGE_DIST_DIR }} ${{ env.optional_build_prod_arguments }}
+          python ./external-builds/pytorch/build_prod_wheels.py \
+            build \
+            --install-rocm \
+            --index-url "https://${{ inputs.cloudfront_url }}/${{ inputs.amdgpu_family }}/" \
+            --pytorch-dir ${{ env.CHECKOUT_ROOT }}/torch \
+            --clean \
+            --output-dir ${{ env.PACKAGE_DIST_DIR }} ${{ env.optional_build_prod_arguments }}

[torch][windows] Build Windows release PyTorch wheels. #994

[torch][windows] Build Windows release PyTorch wheels. #994

Uh oh!

Conversation

ScottTodd commented Jul 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stellaraccident left a comment

Choose a reason for hiding this comment

Uh oh!

ScottTodd commented Jul 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ScottTodd commented Jul 10, 2025

Uh oh!

ScottTodd commented Jul 10, 2025

Uh oh!

marbre left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ScottTodd Jul 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Nem404 commented Jul 10, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ScottTodd commented Jul 9, 2025 •

edited

Loading

ScottTodd commented Jul 10, 2025 •

edited

Loading

ScottTodd Jul 10, 2025 •

edited

Loading