[draft] cuda 124 arm wheel test #124112

tinglvv · 2024-04-15T22:03:46Z

Test pytorch/builder#1775 (merged) for ARM wheel addition
Test pytorch/builder#1804 (WIP) for setting MAX_JOBS

pytorch-bot · 2024-04-15T22:03:49Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/124112

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 10 New Failures

As of commit 339f679 with merge base e04c7b1 ():

NEW FAILURES - The following jobs have failed:

linux-aarch64-binary-manywheel / manywheel-py3_10-cuda-aarch64-test / test (gh)
Process completed with exit code 1.
linux-aarch64-binary-manywheel / manywheel-py3_11-cuda-aarch64-test / test (gh)
Process completed with exit code 1.
linux-aarch64-binary-manywheel / manywheel-py3_12-cuda-aarch64-test / test (gh)
Process completed with exit code 1.
linux-aarch64-binary-manywheel / manywheel-py3_8-cuda-aarch64-test / test (gh)
Process completed with exit code 1.
linux-aarch64-binary-manywheel / manywheel-py3_9-cuda-aarch64-test / test (gh)
Process completed with exit code 1.
linux-binary-manywheel / manywheel-py3_10-cuda12_4-test / test (gh)
RuntimeError: CUDA error: no kernel image is available for execution on the device
linux-binary-manywheel / manywheel-py3_11-cuda12_4-test / test (gh)
RuntimeError: CUDA error: no kernel image is available for execution on the device
linux-binary-manywheel / manywheel-py3_12-cuda12_4-test / test (gh)
RuntimeError: CUDA error: no kernel image is available for execution on the device
linux-binary-manywheel / manywheel-py3_8-cuda12_4-test / test (gh)
RuntimeError: CUDA error: no kernel image is available for execution on the device
linux-binary-manywheel / manywheel-py3_9-cuda12_4-test / test (gh)
RuntimeError: CUDA error: no kernel image is available for execution on the device

This comment was automatically generated by Dr. CI and updates every 15 minutes.

pytorch-bot · 2024-04-15T22:13:51Z

Please seek CI approval before scheduling CIFlow labels

ptrblck · 2024-04-15T23:33:38Z

I believe the origin is set to https://github.com/pytorch/builder, which does not contain your branch, so you would need to change it or create a branch in pytorch/builder instead of your fork.

tinglvv · 2024-04-22T20:29:27Z

@pytorchbot rebase

pytorchmergebot · 2024-04-22T20:31:00Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

pytorchmergebot · 2024-04-22T20:31:06Z

Successfully rebased cuda-124-arm-test onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout cuda-124-arm-test && git pull --rebase)

tinglvv · 2024-04-25T21:48:27Z

@pytorchbot rebase

pytorchmergebot · 2024-04-25T21:49:56Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

pytorchmergebot · 2024-04-25T21:50:02Z

Successfully rebased cuda-124-arm-test onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout cuda-124-arm-test && git pull --rebase)

tinglvv · 2024-05-03T05:06:32Z

@pytorchbot rebase

pytorchmergebot · 2024-05-03T05:08:14Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

pytorchmergebot · 2024-05-03T05:08:16Z

Rebase failed due to Command git -C /home/runner/work/pytorch/pytorch rebase refs/remotes/origin/viable/strict pull/124112/head returned non-zero exit code 1

Rebasing (1/13)
Auto-merging .github/scripts/generate_binary_build_matrix.py
CONFLICT (content): Merge conflict in .github/scripts/generate_binary_build_matrix.py
error: could not apply 44f9c4c57e9... cuda 124 arm wheel test
hint: Resolve all conflicts manually, mark them as resolved with
hint: "git add/rm <conflicted_files>", then run "git rebase --continue".
hint: You can instead skip this commit: run "git rebase --skip".
hint: To abort and get back to the state before "git rebase", run "git rebase --abort".
Could not apply 44f9c4c57e9... cuda 124 arm wheel test

Raised by https://github.com/pytorch/pytorch/actions/runs/8934362376

atalman · 2024-05-10T14:27:59Z

.github/scripts/generate_binary_build_matrix.py

@@ -192,6 +198,7 @@ def translate_desired_cuda(gpu_arch_type: str, gpu_arch_version: str) -> str:
        "cpu-aarch64": "cpu",
        "cpu-cxx11-abi": "cpu-cxx11-abi",
        "cuda": f"cu{gpu_arch_version.replace('.', '')}",
+        "cuda-aarch64": "12.4",


Same as line above this probably needs to be cu124

atalman · 2024-05-10T14:29:14Z

.github/workflows/generated-linux-binary-conda-nightly.yml

@@ -222,6 +222,69 @@ jobs:
      conda-pytorchbot-token-test: ${{ secrets.CONDA_PYTORCHBOT_TOKEN_TEST }}
    uses: ./.github/workflows/_binary-upload.yml

+  conda-py3_8-cuda12_4-build:


This should not be changing conda nightly files, windows workflows etc.. Can you rebase please ? it should solve this issue.

rebasing #124112. too many conflict files, so starting a new PR. Test pytorch/builder#1775 (merged) for ARM wheel addition Test pytorch/builder#1828 (merged) for setting MAX_JOBS Current issue to follow up: #126980 Co-authored-by: Aidyn-A <aidyn.b.aitzhan@gmail.com> Pull Request resolved: #126174 Approved by: https://github.com/nWEIdia, https://github.com/atalman

rebasing pytorch#124112. too many conflict files, so starting a new PR. Test pytorch/builder#1775 (merged) for ARM wheel addition Test pytorch/builder#1828 (merged) for setting MAX_JOBS Current issue to follow up: pytorch#126980 Co-authored-by: Aidyn-A <aidyn.b.aitzhan@gmail.com> Pull Request resolved: pytorch#126174 Approved by: https://github.com/nWEIdia, https://github.com/atalman

pytorch-bot bot added the topic: not user facing topic category label Apr 15, 2024

tinglvv changed the title ~~cuda 124 arm wheel test~~ [draft] cuda 124 arm wheel test Apr 15, 2024

Aidyn-A added the ciflow/binaries_wheel Trigger binary build and upload jobs for wheel on the PR label Apr 15, 2024

pytorch-bot bot removed the ciflow/binaries_wheel Trigger binary build and upload jobs for wheel on the PR label Apr 15, 2024

pytorchbot added the open source label Apr 15, 2024

Aidyn-A added the ciflow/binaries_wheel Trigger binary build and upload jobs for wheel on the PR label Apr 15, 2024

pytorchmergebot force-pushed the cuda-124-arm-test branch from 12b977b to 7f7febb Compare April 22, 2024 20:31

tinglvv added 9 commits April 25, 2024 21:49

cuda 124 arm wheel test

44f9c4c

no extra pytorch install requirements for arm

097e062

change origin to fork

7dff7b8

generate workflow

e77f8da

fix manywheel matrix

5b84f38

fix lint

9a41c6f

update to main as PR is merged

da9c1a5

run regenerate.sh

baf2e93

correct docker image name

f6ec6d3

pytorchmergebot force-pushed the cuda-124-arm-test branch from 21c28a1 to f6ec6d3 Compare April 25, 2024 21:50

tinglvv added 4 commits April 25, 2024 16:00

correct tag

10e94ce

run regenerate.sh

81b80d8

change origin

3930312

test increase timeout

0b0ce9d

tinglvv added 2 commits May 5, 2024 01:01

trigger test

6f01992

touch trigger

e6e4780

tinglvv mentioned this pull request May 7, 2024

OOM error in building ARM wheel pytorch/builder#1812

Open

tinglvv added 8 commits May 7, 2024 02:16

trigger test

b098d03

trigger test for removing openblas.so

4542c85

trigger test (max_job=6) (hopper only)

5aafdb8

trigger test (no error message last time)

e056535

trigger with maxjobs=4 and 9.0

c044db5

cherrypick builder pytorch#1808

dd880e5

4 and hopper

f44863a

max_jobs=5 and corrected path

339f679

atalman reviewed May 10, 2024

View reviewed changes

This was referenced May 10, 2024

[draft][rebase]cuda 124 arm wheel test #125934

Closed

CUDA 12.4 ARM wheel integration to CD - nightly build #126174

Closed

tinglvv mentioned this pull request May 23, 2024

Long build compilation time (>5 hours) for CUDA ARM build #126980

Open

tinglvv closed this May 27, 2024

tinglvv deleted the cuda-124-arm-test branch May 27, 2024 06:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[draft] cuda 124 arm wheel test #124112

[draft] cuda 124 arm wheel test #124112

tinglvv commented Apr 15, 2024 •

edited

Loading

pytorch-bot bot commented Apr 15, 2024 •

edited

Loading

pytorch-bot bot commented Apr 15, 2024

ptrblck commented Apr 15, 2024

tinglvv commented Apr 22, 2024

pytorchmergebot commented Apr 22, 2024

pytorchmergebot commented Apr 22, 2024

tinglvv commented Apr 25, 2024

pytorchmergebot commented Apr 25, 2024

pytorchmergebot commented Apr 25, 2024

tinglvv commented May 3, 2024

pytorchmergebot commented May 3, 2024

pytorchmergebot commented May 3, 2024

atalman May 10, 2024

atalman May 10, 2024 •

edited

Loading

[draft] cuda 124 arm wheel test #124112

[draft] cuda 124 arm wheel test #124112

Conversation

tinglvv commented Apr 15, 2024 • edited Loading

pytorch-bot bot commented Apr 15, 2024 • edited Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/124112

❌ 10 New Failures

pytorch-bot bot commented Apr 15, 2024

ptrblck commented Apr 15, 2024

tinglvv commented Apr 22, 2024

pytorchmergebot commented Apr 22, 2024

pytorchmergebot commented Apr 22, 2024

tinglvv commented Apr 25, 2024

pytorchmergebot commented Apr 25, 2024

pytorchmergebot commented Apr 25, 2024

tinglvv commented May 3, 2024

pytorchmergebot commented May 3, 2024

pytorchmergebot commented May 3, 2024

atalman May 10, 2024

Choose a reason for hiding this comment

atalman May 10, 2024 • edited Loading

Choose a reason for hiding this comment

tinglvv commented Apr 15, 2024 •

edited

Loading

pytorch-bot bot commented Apr 15, 2024 •

edited

Loading

atalman May 10, 2024 •

edited

Loading