CUDA 12.4 ARM wheel integration to CD - nightly build #126174

tinglvv · 2024-05-14T15:23:01Z

rebasing #124112.
too many conflict files, so starting a new PR.

Test pytorch/builder#1775 (merged) for ARM wheel addition
Test pytorch/builder#1828 (merged) for setting MAX_JOBS

Current issue to follow up:
#126980

cc @atalman @malfet @ptrblck @nWEIdia @Aidyn-A

pytorch-bot · 2024-05-14T15:23:04Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/126174

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (2 Unrelated Failures)

As of commit 42bb184 with merge base f0366de ():

UNSTABLE - The following jobs failed but were likely due to flakiness present on trunk and has been marked as unstable:

pull / linux-focal-cuda12.4-py3.10-gcc9 / build (gh) (#127108)
/var/lib/jenkins/workspace/aten/src/ATen/cuda/CUDASparseDescriptors.h:119:68: error: ‘cusparseStatus_t cusparseCreateBsrsm2Info(bsrsm2Info**)’ is deprecated: The routine will be removed in the next major release [-Werror=deprecated-declarations]
pull / linux-focal-cuda12.4-py3.10-gcc9-sm86 / build (gh) (#127104)
/var/lib/jenkins/workspace/aten/src/ATen/cuda/CUDASparseDescriptors.h:119:68: error: ‘cusparseStatus_t cusparseCreateBsrsm2Info(bsrsm2Info**)’ is deprecated: The routine will be removed in the next major release [-Werror=deprecated-declarations]

This comment was automatically generated by Dr. CI and updates every 15 minutes.

tinglvv · 2024-05-16T02:19:40Z

@pytorchbot rebase

pytorchmergebot · 2024-05-16T02:21:23Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

pytorchmergebot · 2024-05-16T02:21:27Z

Successfully rebased cuda124-arm-ci onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout cuda124-arm-ci && git pull --rebase)

.github/workflows/generated-linux-aarch64-binary-manywheel-nightly.yml

.github/scripts/generate_binary_build_matrix.py

Trying to explore options to help land: pytorch/pytorch#126174 Current [manywheel-py3_9-cuda-aarch64-build / build](https://github.com/pytorch/pytorch/actions/runs/9112985273/job/25053413689?pr=126174#logs) Takes around 6hrs (building only sm90 arch). Hence trying to bring slightly bigger worker, see if we can bring build time to manageable time 3-3.5hrs.

tinglvv · 2024-05-17T11:59:10Z

@pytorchbot rebase

pytorchmergebot · 2024-05-17T12:02:27Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

pytorchmergebot · 2024-05-17T12:02:32Z

Successfully rebased cuda124-arm-ci onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout cuda124-arm-ci && git pull --rebase)

.github/workflows/_binary-test-linux.yml

.github/workflows/generated-linux-aarch64-binary-manywheel-nightly.yml

README.md

nWEIdia

Changes look great!

.github/workflows/_binary-build-linux.yml

.github/workflows/generated-linux-binary-conda-nightly.yml

atalman · 2024-05-23T12:39:31Z

.github/templates/linux_binary_build_workflow.yml.j2

@@ -71,12 +71,19 @@ jobs:
      {%- if config.pytorch_extra_install_requirements is defined and config.pytorch_extra_install_requirements|d('')|length > 0  %}
      PYTORCH_EXTRA_INSTALL_REQUIREMENTS: !{{ config.pytorch_extra_install_requirements }}
      {%- endif %}
+      {%- if config["gpu_arch_type"] == "cuda-aarch64" %}


This approach is good, just need to remove else condition with timeout minutes 210

tinglvv · 2024-05-23T13:31:32Z

@pytorchbot rebase

tinglvv · 2024-05-24T06:11:59Z

@pytorchbot merge -i

pytorchmergebot · 2024-05-24T06:13:52Z

Merge started

Your change will be merged while ignoring the following 2 checks: windows-binary-wheel / wheel-py3_8-cpu-test, windows-binary-conda / conda-py3_8-cuda12_1-test

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

tinglvv · 2024-05-24T06:14:21Z

the 3-10 aarch64 failure is due to timeout (6 hrs limit), could be due to network condition, can increase the timeout-minutes accordingly later. Merging as all the rest aarch64 builds are passing. Windows build failure is unrelated.

pytorchmergebot · 2024-05-24T11:57:02Z

Merge failed

Reason: 1 jobs have failed, first few of them are: linux-aarch64-binary-manywheel / manywheel-py3_11-cuda-aarch64-build / build

Details for Dev Infra team

Raised by workflow job

tinglvv · 2024-05-24T20:29:12Z

@pytorchbot rebase

pytorchmergebot · 2024-05-24T20:30:41Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

pytorchmergebot · 2024-05-24T20:30:42Z

Rebase failed due to Command git -C /home/runner/work/pytorch/pytorch rebase refs/remotes/origin/viable/strict pull/126174/head returned non-zero exit code 1

Rebasing (1/34)
Rebasing (2/34)
Rebasing (3/34)
Rebasing (4/34)
Rebasing (5/34)
Rebasing (6/34)
Rebasing (7/34)
Rebasing (8/34)
Rebasing (9/34)
Rebasing (10/34)
Rebasing (11/34)
Auto-merging .github/workflows/generated-linux-binary-manywheel-main.yml
CONFLICT (content): Merge conflict in .github/workflows/generated-linux-binary-manywheel-main.yml
error: could not apply 5007c3953c3... move up gpu arch type check to above -test definition
hint: Resolve all conflicts manually, mark them as resolved with
hint: "git add/rm <conflicted_files>", then run "git rebase --continue".
hint: You can instead skip this commit: run "git rebase --skip".
hint: To abort and get back to the state before "git rebase", run "git rebase --abort".
hint: Disable this message with "git config advice.mergeConflict false"
Could not apply 5007c3953c3... move up gpu arch type check to above -test definition

Raised by https://github.com/pytorch/pytorch/actions/runs/9229584549

tinglvv · 2024-05-26T12:47:51Z

@pytorchbot merge -i

pytorchmergebot · 2024-05-26T12:49:46Z

Merge started

Your change will be merged while ignoring the following 0 checks:

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2024-05-26T18:48:19Z

The merge job was canceled or timed out. This most often happen if two merge requests were issued for the same PR, or if merge job was waiting for more than 6 hours for tests to finish. In later case, please do not hesitate to reissue the merge command
For more information see pytorch-bot wiki.

tinglvv · 2024-05-27T05:47:45Z

@pytorchbot merge

pytorchmergebot · 2024-05-27T05:49:59Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

rebasing pytorch#124112. too many conflict files, so starting a new PR. Test pytorch/builder#1775 (merged) for ARM wheel addition Test pytorch/builder#1828 (merged) for setting MAX_JOBS Current issue to follow up: pytorch#126980 Co-authored-by: Aidyn-A <aidyn.b.aitzhan@gmail.com> Pull Request resolved: pytorch#126174 Approved by: https://github.com/nWEIdia, https://github.com/atalman

pytorch-bot bot added the topic: not user facing topic category label May 14, 2024

pytorchbot added the open source label May 14, 2024

Aidyn-A added the ciflow/binaries Trigger all binary build and upload jobs on the PR label May 14, 2024

pytorchmergebot force-pushed the cuda124-arm-ci branch from f8dc9c3 to e717e5a Compare May 16, 2024 02:21

atalman reviewed May 16, 2024

View reviewed changes

.github/workflows/generated-linux-aarch64-binary-manywheel-nightly.yml Outdated Show resolved Hide resolved

atalman reviewed May 16, 2024

View reviewed changes

.github/workflows/generated-linux-aarch64-binary-manywheel-nightly.yml Show resolved Hide resolved

atalman reviewed May 16, 2024

View reviewed changes

.github/scripts/generate_binary_build_matrix.py Outdated Show resolved Hide resolved

atalman mentioned this pull request May 16, 2024

Add m7g.2xlarge workers for linux-aarch64 gpu builds pytorch/test-infra#5210

Merged

pytorchmergebot force-pushed the cuda124-arm-ci branch from 503eaca to d2cdca7 Compare May 17, 2024 12:02

nWEIdia reviewed May 21, 2024

View reviewed changes

.github/workflows/_binary-test-linux.yml Outdated Show resolved Hide resolved

nWEIdia reviewed May 21, 2024

View reviewed changes

.github/workflows/generated-linux-aarch64-binary-manywheel-nightly.yml Outdated Show resolved Hide resolved

tinglvv marked this pull request as ready for review May 22, 2024 01:30

tinglvv requested a review from a team as a code owner May 22, 2024 01:30

tinglvv changed the title ~~[draft] [rebase] cuda 124 arm wheel test~~ CUDA 12.4 ARM wheel integration to CI May 22, 2024

nWEIdia reviewed May 22, 2024

View reviewed changes

README.md Outdated Show resolved Hide resolved

nWEIdia approved these changes May 22, 2024

View reviewed changes

nWEIdia reviewed May 22, 2024

View reviewed changes

.github/workflows/_binary-build-linux.yml Show resolved Hide resolved

nWEIdia mentioned this pull request May 22, 2024

[Do not merge] Test for "Cuda124 arm ci" #126910

Closed

atalman reviewed May 23, 2024

View reviewed changes

.github/workflows/_binary-build-linux.yml Show resolved Hide resolved

atalman reviewed May 23, 2024

View reviewed changes

.github/workflows/generated-linux-binary-conda-nightly.yml Outdated Show resolved Hide resolved

atalman reviewed May 23, 2024

View reviewed changes

atalman approved these changes May 23, 2024

View reviewed changes

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label May 24, 2024

pytorchmergebot added the merging label May 24, 2024

pytorchmergebot removed the merging label May 24, 2024

set timeout-minutes to 420 for cuda-aarch64

3adde97

This was referenced May 24, 2024

Cache OpenBLAS build to CUDA ARM Docker image pytorch/builder#1833

Merged

Enable linker script optimization for CUDA ARM wheel pytorch/builder#1832

Merged

tinglvv added 2 commits May 24, 2024 12:27

upgrade builder to m7g.4xlarge

c1ad06e

test linker script (tinglvv/builder encore)

090c907

Aidyn-A and others added 2 commits May 24, 2024 14:37

re-trigger CI

8f729af

set back to main branch for builder repo

42bb184

pytorchmergebot added the merging label May 26, 2024

pytorchmergebot closed this in 1c2e221 May 27, 2024

pytorchmergebot added Merged and removed merging labels May 27, 2024

tinglvv mentioned this pull request May 29, 2024

add cuda version to arm nightly wheel pytorch/builder#1838

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CUDA 12.4 ARM wheel integration to CD - nightly build #126174

CUDA 12.4 ARM wheel integration to CD - nightly build #126174

tinglvv commented May 14, 2024 •

edited

pytorch-bot bot commented May 14, 2024 •

edited

tinglvv commented May 16, 2024

pytorchmergebot commented May 16, 2024

pytorchmergebot commented May 16, 2024

tinglvv commented May 17, 2024

pytorchmergebot commented May 17, 2024

pytorchmergebot commented May 17, 2024

nWEIdia left a comment

atalman May 23, 2024

tinglvv commented May 23, 2024

tinglvv commented May 24, 2024

pytorchmergebot commented May 24, 2024

tinglvv commented May 24, 2024

pytorchmergebot commented May 24, 2024

tinglvv commented May 24, 2024

pytorchmergebot commented May 24, 2024

pytorchmergebot commented May 24, 2024

tinglvv commented May 26, 2024

pytorchmergebot commented May 26, 2024

pytorchmergebot commented May 26, 2024

tinglvv commented May 27, 2024

pytorchmergebot commented May 27, 2024

CUDA 12.4 ARM wheel integration to CD - nightly build #126174

CUDA 12.4 ARM wheel integration to CD - nightly build #126174

Conversation

tinglvv commented May 14, 2024 • edited

pytorch-bot bot commented May 14, 2024 • edited

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/126174

✅ You can merge normally! (2 Unrelated Failures)

tinglvv commented May 16, 2024

pytorchmergebot commented May 16, 2024

pytorchmergebot commented May 16, 2024

tinglvv commented May 17, 2024

pytorchmergebot commented May 17, 2024

pytorchmergebot commented May 17, 2024

nWEIdia left a comment

Choose a reason for hiding this comment

atalman May 23, 2024

Choose a reason for hiding this comment

tinglvv commented May 23, 2024

tinglvv commented May 24, 2024

pytorchmergebot commented May 24, 2024

Merge started

tinglvv commented May 24, 2024

pytorchmergebot commented May 24, 2024

Merge failed

tinglvv commented May 24, 2024

pytorchmergebot commented May 24, 2024

pytorchmergebot commented May 24, 2024

tinglvv commented May 26, 2024

pytorchmergebot commented May 26, 2024

Merge started

pytorchmergebot commented May 26, 2024

tinglvv commented May 27, 2024

pytorchmergebot commented May 27, 2024

Merge started

tinglvv commented May 14, 2024 •

edited

pytorch-bot bot commented May 14, 2024 •

edited