Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[tp] fix torch compile regression #111521

Closed
wants to merge 3 commits into from
Closed

[tp] fix torch compile regression #111521

wants to merge 3 commits into from

Conversation

wanchaol
Copy link
Contributor

@wanchaol wanchaol commented Oct 18, 2023 •
edited

Stack from ghstack (oldest at bottom):

The most recent refactor of TP
#111160 breaks torch compile
path, so reverting the behavior back by:

  1. use the old default prepare_input/output
  2. add the colwise/rowwise parallel test instead

@wanchaol
[tp] fix torch compile regression
a286053 
The most recent refactor of TP
#111160 breaks torch compile
path, so reverting the behavior back by:
1. use the old default prepare_input/output
2. add the colwise/rowwise parallel test instead

[ghstack-poisoned]
@pytorch-bot
Copy link

pytorch-bot bot commented Oct 18, 2023 •
edited

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/111521

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (3 Unrelated Failures)

As of commit 7f29dec with merge base 5c39552 (image):

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

UNSTABLE - The following job failed but was likely due to flakiness present on trunk and has been marked as unstable:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

wanchaol added a commit that referenced this pull request Oct 18, 2023
@wanchaol
[tp] fix torch compile regression
b6a5a74 
The most recent refactor of TP
#111160 breaks torch compile
path, so reverting the behavior back by:
1. use the old default prepare_input/output
2. add the colwise/rowwise parallel test instead

ghstack-source-id: 4b9faae80116d25f5525b99e651985e14691aeab
Pull Request resolved: #111521
fduwjj
fduwjj reviewed Oct 18, 2023
View reviewed changes
torch/distributed/tensor/parallel/style.py Outdated
Comment on lines 485 to 486
_prepare_input=None,
_prepare_output=None,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But this somehow revert the input_layouts and output_layouts? So does this mean when users want to use input_layouts and output_layouts, they have to set _prepare_input to None?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is just a quick fix, I'll do a big rewrite on top to make sure user behavior won't change

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK I gave up on the big rewrite and found a quicker solution right now, however it does not fundamentally fix the torch.compile problem in the new style input_layouts and output_layouts path (i.e. to configure sequence parallel torch.compile still fail). We should do a full rewrite soon to make the TP code cleaner and tracing friendly

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this "hack" preserves the intended behavior, but I dislike it..

@wanchaol
Update on "[tp] fix torch compile regression"
5d92ebd 
The most recent refactor of TP
#111160 breaks torch compile
path, so reverting the behavior back by:
1. use the old default prepare_input/output
2. add the colwise/rowwise parallel test instead

[ghstack-poisoned]
wanchaol added a commit that referenced this pull request Oct 19, 2023
@wanchaol
[tp] fix torch compile regression
a1bf3c5 
The most recent refactor of TP
#111160 breaks torch compile
path, so reverting the behavior back by:
1. use the old default prepare_input/output
2. add the colwise/rowwise parallel test instead

ghstack-source-id: 0865d507b2debbd4bfa708a0ff7465a23e64f5de
Pull Request resolved: #111521
@wanchaol wanchaol added ciflow/trunk Trigger trunk jobs on your pull request release notes: distributed (dtensor) release notes category labels Oct 19, 2023
@wanchaol
Copy link
Contributor Author

@pytorchbot merge

@pytorchmergebot
Copy link
Collaborator

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@wanchaol wanchaol added the ciflow/periodic Trigger jobs ran periodically on master (periodic.yml) on the PR label Oct 19, 2023
@pytorchmergebot
Copy link
Collaborator

Merge failed

Reason: 1 mandatory check(s) failed. The first few are:

Dig deeper by viewing the failures on hud

Details for Dev Infra team Raised by workflow job

Failing merge rule: Core Maintainers

@wanchaol
Update on "[tp] fix torch compile regression"
7f29dec 
The most recent refactor of TP
#111160 breaks torch compile
path, so reverting the behavior back by:
1. use the old default prepare_input/output
2. add the colwise/rowwise parallel test instead

[ghstack-poisoned]
wanchaol added a commit that referenced this pull request Oct 19, 2023
@wanchaol
[tp] fix torch compile regression
476ea2f 
The most recent refactor of TP
#111160 breaks torch compile
path, so reverting the behavior back by:
1. use the old default prepare_input/output
2. add the colwise/rowwise parallel test instead

ghstack-source-id: 2dbb05c7bb3012e67af3e07778032ed1b52cc69d
Pull Request resolved: #111521
@wanchaol
Copy link
Contributor Author

@pytorchbot merge

@pytorchmergebot
Copy link
Collaborator

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@facebook-github-bot facebook-github-bot deleted the gh/wanchaol/376/head branch October 22, 2023 14:23
xuhancn pushed a commit to xuhancn/pytorch that referenced this pull request Nov 7, 2023
@wanchaol @xuhancn
[tp] fix torch compile regression (pytorch#111521)
a7b80a3 
The most recent refactor of TP
pytorch#111160 breaks torch compile
path, so reverting the behavior back by:
1. use the old default prepare_input/output
2. add the colwise/rowwise parallel test instead
Pull Request resolved: pytorch#111521
Approved by: https://github.com/fduwjj
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@fduwjj fduwjj fduwjj approved these changes

@mrshenli mrshenli Awaiting requested review from mrshenli

@zhaojuanmao zhaojuanmao Awaiting requested review from zhaojuanmao

@rohan-varma rohan-varma Awaiting requested review from rohan-varma

@H-Huang H-Huang Awaiting requested review from H-Huang

@awgu awgu Awaiting requested review from awgu

@kwen2501 kwen2501 Awaiting requested review from kwen2501

@fegin fegin Awaiting requested review from fegin

@wz337 wz337 Awaiting requested review from wz337

@kiukchung kiukchung Awaiting requested review from kiukchung

@d4l3k d4l3k Awaiting requested review from d4l3k

Assignees
No one assigned
Labels
ciflow/periodic Trigger jobs ran periodically on master (periodic.yml) on the PR ciflow/trunk Trigger trunk jobs on your pull request Merged release notes: distributed (dtensor) release notes category
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@wanchaol @pytorchmergebot @fduwjj