New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Allow overriding per-dim group options via _MeshEnv.set_dim_group_options #126599

Closed

yifuwang wants to merge 18 commits into gh/yifuwang/85/base from gh/yifuwang/85/head

Contributor

yifuwang commented May 18, 2024 •

edited by pytorch-bot bot

Stack from ghstack (oldest at bottom):

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @penguinwu @fegin @XilunWu @wanchaol @fduwjj @wz337 @tianyu-l @wconstab @yf225 @chauhang @d4l3k


          [WIP] _set_mesh_dim_group_options

d9b4fbf

[ghstack-poisoned]

This was referenced May 18, 2024

Introduce ProcessGroupCudaP2P #122163

Closed

Introduce cuda_p2p based fused_all_gather_matmul and fused_matmul_reduce_scatter #126597

Closed

pytorch-bot bot commented May 18, 2024 •

edited

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/126599

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 4c80244 with merge base d8d0bf2 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

yifuwang mentioned this pull request

Introduce Inductor passes to micro-pipeline all-gather-matmul and matmul-reduce-scatter in certain cases #126598

Closed

pytorch-bot bot added the oncall: distributed label

yifuwang added a commit that referenced this pull request


          [WIP] _set_mesh_dim_group_options

c2db8f4

ghstack-source-id: 3affd43fbf799cc2888289225d36347409d5c3dc
Pull Request resolved: #126599


          Update on "[WIP] _set_mesh_dim_group_options"

98c7a7c

cc mrshenli pritamdamania87 zhaojuanmao satgera gqchen aazzolini osalpekar jiayisuse H-Huang kwen2501 awgu penguinwu fegin XilunWu wanchaol fduwjj wz337 tianyu-l wconstab yf225 chauhang d4l3k

[ghstack-poisoned]

yifuwang added a commit that referenced this pull request


          [WIP] _set_mesh_dim_group_options

8a6b914

ghstack-source-id: b20313fee3f27ad5104290d6a5a44fa4e52d30aa
Pull Request resolved: #126599


          Update on "[WIP] _set_mesh_dim_group_options"

cc mrshenli pritamdamania87 zhaojuanmao satgera gqchen aazzolini osalpekar jiayisuse H-Huang kwen2501 awgu penguinwu fegin XilunWu wanchaol fduwjj wz337 tianyu-l wconstab yf225 chauhang d4l3k

[ghstack-poisoned]

yifuwang added a commit that referenced this pull request


          [WIP] _set_mesh_dim_group_options

7a690f9

ghstack-source-id: 36f90cd502058de8b4278a5f6eb42c638502d9f0
Pull Request resolved: #126599


          Update on "[WIP] _set_mesh_dim_group_options"

841c6a8

cc mrshenli pritamdamania87 zhaojuanmao satgera gqchen aazzolini osalpekar jiayisuse H-Huang kwen2501 awgu penguinwu fegin XilunWu wanchaol fduwjj wz337 tianyu-l wconstab yf225 chauhang d4l3k

[ghstack-poisoned]

yifuwang added a commit that referenced this pull request


          [WIP] _set_mesh_dim_group_options

d7d8d23

ghstack-source-id: ff49b96912ec614b91df08ec4478f5c4bd881cd6
Pull Request resolved: #126599


          Update on "[WIP] _set_mesh_dim_group_options"

1c708fe

cc mrshenli pritamdamania87 zhaojuanmao satgera gqchen aazzolini osalpekar jiayisuse H-Huang kwen2501 awgu penguinwu fegin XilunWu wanchaol fduwjj wz337 tianyu-l wconstab yf225 chauhang d4l3k

[ghstack-poisoned]

yifuwang mentioned this pull request

Introduce cuda_p2p based fused_all_gather_matmul and fused_matmul_reduce_scatter #126634

Closed

yifuwang added a commit that referenced this pull request


          [WIP] _set_mesh_dim_group_options

703dbf7

ghstack-source-id: ff49b96912ec614b91df08ec4478f5c4bd881cd6
Pull Request resolved: #126599


          Update on "[WIP] _set_mesh_dim_group_options"

7e27bfc

cc mrshenli pritamdamania87 zhaojuanmao satgera gqchen aazzolini osalpekar jiayisuse H-Huang kwen2501 awgu penguinwu fegin XilunWu wanchaol fduwjj wz337 tianyu-l wconstab yf225 chauhang d4l3k

[ghstack-poisoned]

yifuwang added a commit that referenced this pull request


          [WIP] _set_mesh_dim_group_options

7a8aa94

ghstack-source-id: 5b2dae763f7d1810f08467f839c64a20e872623c
Pull Request resolved: #126599

wanchaol reviewed

View reviewed changes

torch/distributed/device_mesh.py Outdated Show resolved Hide resolved

torch/distributed/device_mesh.py Outdated Show resolved Hide resolved

torch/distributed/device_mesh.py Show resolved Hide resolved


          Update on "[WIP] _set_mesh_dim_group_options"

8f4ef90

cc mrshenli pritamdamania87 zhaojuanmao satgera gqchen aazzolini osalpekar jiayisuse H-Huang kwen2501 awgu penguinwu fegin XilunWu wanchaol fduwjj wz337 tianyu-l wconstab yf225 chauhang d4l3k

[ghstack-poisoned]

yifuwang added a commit that referenced this pull request


          [WIP] _set_mesh_dim_group_options

842f99a

ghstack-source-id: 651c4bf12ced6104c72eea1c4719d8e16475fdae
Pull Request resolved: #126599


          Update on "[WIP] _set_mesh_dim_group_options"

95d7aaf

cc mrshenli pritamdamania87 zhaojuanmao satgera gqchen aazzolini osalpekar jiayisuse H-Huang kwen2501 awgu penguinwu fegin XilunWu wanchaol fduwjj wz337 tianyu-l wconstab yf225 chauhang d4l3k

[ghstack-poisoned]

yifuwang added a commit that referenced this pull request


          [WIP] _set_mesh_dim_group_options

e93f4c5

ghstack-source-id: 651c4bf12ced6104c72eea1c4719d8e16475fdae
Pull Request resolved: #126599


          Update on "[WIP] _set_mesh_dim_group_options"

80a104d

cc mrshenli pritamdamania87 zhaojuanmao satgera gqchen aazzolini osalpekar jiayisuse H-Huang kwen2501 awgu penguinwu fegin XilunWu wanchaol fduwjj wz337 tianyu-l wconstab yf225 chauhang d4l3k

[ghstack-poisoned]

yifuwang added a commit that referenced this pull request


          [WIP] _set_mesh_dim_group_options

37c5e91

ghstack-source-id: 8ab9ab99138eeb16a242fb63d33e06ed77344e9a
Pull Request resolved: #126599


          Update on "[WIP] _set_mesh_dim_group_options"

0b6a4c8

cc mrshenli pritamdamania87 zhaojuanmao satgera gqchen aazzolini osalpekar jiayisuse H-Huang kwen2501 awgu penguinwu fegin XilunWu wanchaol fduwjj wz337 tianyu-l wconstab yf225 chauhang d4l3k

[ghstack-poisoned]

yifuwang added a commit that referenced this pull request


          [WIP] _set_mesh_dim_group_options

3f42b55

ghstack-source-id: dfb90df6f9eab666c9e65d673b74bafee2736dfd
Pull Request resolved: #126599


          Update on "[WIP] _set_mesh_dim_group_options"

5ed025f

cc mrshenli pritamdamania87 zhaojuanmao satgera gqchen aazzolini osalpekar jiayisuse H-Huang kwen2501 awgu penguinwu fegin XilunWu wanchaol fduwjj wz337 tianyu-l wconstab yf225 chauhang d4l3k

[ghstack-poisoned]

yifuwang added a commit that referenced this pull request


          [WIP] _set_mesh_dim_group_options

6f9c45e

ghstack-source-id: 0324a1947296112e37469d64138cffe6285ee644
Pull Request resolved: #126599


          Update on "[WIP] _set_mesh_dim_group_options"

a4f3d34

cc mrshenli pritamdamania87 zhaojuanmao satgera gqchen aazzolini osalpekar jiayisuse H-Huang kwen2501 awgu penguinwu fegin XilunWu wanchaol fduwjj wz337 tianyu-l wconstab yf225 chauhang d4l3k

[ghstack-poisoned]

This was referenced May 29, 2024

force_stride_order on fused_all_gather_matmul/fused_matmul_reduce_scatter's operands to avoid a copy due to layout transformation #127454

Open

Improve the scheduling for fused_matmul_reduce_scatter #127455

Open


          Update on "Allow overriding per-dim group options via _MeshEnv.set_di…

1a03d53

…m_group_options"

[ghstack-poisoned]

yifuwang changed the title ~~[WIP] _set_mesh_dim_group_options~~ Allow overriding per-dim group options via _MeshEnv.set_dim_group_options


          Update on "Allow overriding per-dim group options via _MeshEnv.set_di…

e2f94b3

…m_group_options"

[ghstack-poisoned]

yifuwang requested a review from wanchaol

May 29, 2024 23:02

yifuwang marked this pull request as ready for review

May 29, 2024 23:02

yifuwang mentioned this pull request

Add profiler annotation for fused_all_gather_matmul and fused_matmul_reduce_scatter #127556

Open

yifuwang added 2 commits

May 30, 2024 11:52


          Update on "Allow overriding per-dim group options via _MeshEnv.set_di…

f7a72dd

…m_group_options"

[ghstack-poisoned]


          Update on "Allow overriding per-dim group options via _MeshEnv.set_di…

a96de27

…m_group_options"

[ghstack-poisoned]

yifuwang mentioned this pull request

Draft

yifuwang added 2 commits

June 3, 2024 11:08


          Update on "Allow overriding per-dim group options via _MeshEnv.set_di…

479778c

…m_group_options"

[ghstack-poisoned]


          Update on "Allow overriding per-dim group options via _MeshEnv.set_di…

4c80244

…m_group_options"

[ghstack-poisoned]

wanchaol approved these changes

View reviewed changes

Contributor

wanchaol left a comment

looks great! thanks!

Contributor Author

yifuwang commented Jun 6, 2024

@pytorchbot merge

pytorch-bot bot added the ciflow/trunk label

pytorchmergebot added the merging label

Collaborator

pytorchmergebot commented Jun 6, 2024

Merge failed

Reason: This PR needs a release notes: label
If your changes are user facing and intended to be a part of release notes, please use a label starting with release notes:.

If not, please add the topic: not user facing label.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "topic: not user facing"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

Details for Dev Infra team

Raised by workflow job

pytorchmergebot removed the merging label

yifuwang added the topic: not user facing label

Contributor Author

yifuwang commented Jun 6, 2024

@pytorchbot merge

pytorchmergebot added the merging label

Collaborator

pytorchmergebot commented Jun 6, 2024

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot added the Merged label

pytorchmergebot closed this in

1d0c108

pytorchmergebot removed the merging label

yifuwang mentioned this pull request

[IntraNodeComm] switch cudaIpc APIs to cuMem APIs #128164

Open

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment