Skip to content

[dtensor] add StrategyType and TupleStrategy #99435

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 12 commits into from
Closed

[dtensor] add StrategyType and TupleStrategy #99435

wants to merge 12 commits into from

Conversation

@wanchaol
Copy link
Collaborator

@wanchaol wanchaol commented Apr 18, 2023 •
edited
Loading

Stack from ghstack (oldest at bottom):

This PR refactors the current StrategyList. It introduces a
StrategyType, which is the base class of Strategy, and it have
two sub strategies:

  1. Refactor the previous StrategyList to OpStrategy
  2. Add TupleStrategy, the new strategy added to deal with tuple cases where
    it could return multiple different OpStrategy for an op.

This would help support a more complicated op and unblocks compile mode
FSDP

@wanchaol
[dtensor] add StrategyType and TupleStrategy
0699ef2 
This PR refactors the current StrategyList. It introduces a
StrategyType, which is the base class of Strategy, and it have
two sub strategies:

1. Refactor the previous StrategyList to OpStrategy
2. Add TupleStrategy, the new strategy added to deal with tuple cases where
it could return multiple different OpStrategy for an op.

This would help support a more complicated op and unblocks compile mode
FSDP

[ghstack-poisoned]
@pytorch-bot
Copy link

pytorch-bot bot commented Apr 18, 2023 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/99435

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

✅ No Failures

As of commit f07da91:
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

This was referenced Apr 18, 2023
Closed
Closed
Closed
Closed
Closed
@wanchaol wanchaol added the release notes: distributed (dtensor) release notes category label Apr 18, 2023
@wanchaol
Update on "[dtensor] add StrategyType and TupleStrategy"
de450fa 
This PR refactors the current StrategyList. It introduces a
StrategyType, which is the base class of Strategy, and it have
two sub strategies:

1. Refactor the previous StrategyList to OpStrategy
2. Add TupleStrategy, the new strategy added to deal with tuple cases where
it could return multiple different OpStrategy for an op.

This would help support a more complicated op and unblocks compile mode
FSDP

[ghstack-poisoned]
@wanchaol wanchaol mentioned this pull request Apr 19, 2023
Closed
mrshenli
mrshenli approved these changes Apr 20, 2023
View reviewed changes
wanchaol added 10 commits April 20, 2023 22:05
@wanchaol
Update on "[dtensor] add StrategyType and TupleStrategy"
7b2c2c7 
This PR refactors the current StrategyList. It introduces a
StrategyType, which is the base class of Strategy, and it have
two sub strategies:

1. Refactor the previous StrategyList to OpStrategy
2. Add TupleStrategy, the new strategy added to deal with tuple cases where
it could return multiple different OpStrategy for an op.

This would help support a more complicated op and unblocks compile mode
FSDP

[ghstack-poisoned]
@wanchaol
Update on "[dtensor] add StrategyType and TupleStrategy"
2d8a228 
This PR refactors the current StrategyList. It introduces a
StrategyType, which is the base class of Strategy, and it have
two sub strategies:

1. Refactor the previous StrategyList to OpStrategy
2. Add TupleStrategy, the new strategy added to deal with tuple cases where
it could return multiple different OpStrategy for an op.

This would help support a more complicated op and unblocks compile mode
FSDP

[ghstack-poisoned]
@wanchaol
Update on "[dtensor] add StrategyType and TupleStrategy"
cbb342a 
This PR refactors the current StrategyList. It introduces a
StrategyType, which is the base class of Strategy, and it have
two sub strategies:

1. Refactor the previous StrategyList to OpStrategy
2. Add TupleStrategy, the new strategy added to deal with tuple cases where
it could return multiple different OpStrategy for an op.

This would help support a more complicated op and unblocks compile mode
FSDP

[ghstack-poisoned]
@wanchaol
Update on "[dtensor] add StrategyType and TupleStrategy"
db2b48e 
This PR refactors the current StrategyList. It introduces a
StrategyType, which is the base class of Strategy, and it have
two sub strategies:

1. Refactor the previous StrategyList to OpStrategy
2. Add TupleStrategy, the new strategy added to deal with tuple cases where
it could return multiple different OpStrategy for an op.

This would help support a more complicated op and unblocks compile mode
FSDP

[ghstack-poisoned]
@wanchaol
Update on "[dtensor] add StrategyType and TupleStrategy"
d27d70a 
This PR refactors the current StrategyList. It introduces a
StrategyType, which is the base class of Strategy, and it have
two sub strategies:

1. Refactor the previous StrategyList to OpStrategy
2. Add TupleStrategy, the new strategy added to deal with tuple cases where
it could return multiple different OpStrategy for an op.

This would help support a more complicated op and unblocks compile mode
FSDP

[ghstack-poisoned]
@wanchaol
Update on "[dtensor] add StrategyType and TupleStrategy"
0e3fe7d 
This PR refactors the current StrategyList. It introduces a
StrategyType, which is the base class of Strategy, and it have
two sub strategies:

1. Refactor the previous StrategyList to OpStrategy
2. Add TupleStrategy, the new strategy added to deal with tuple cases where
it could return multiple different OpStrategy for an op.

This would help support a more complicated op and unblocks compile mode
FSDP

[ghstack-poisoned]
@wanchaol
Update on "[dtensor] add StrategyType and TupleStrategy"
ba59ca6 
This PR refactors the current StrategyList. It introduces a
StrategyType, which is the base class of Strategy, and it have
two sub strategies:

1. Refactor the previous StrategyList to OpStrategy
2. Add TupleStrategy, the new strategy added to deal with tuple cases where
it could return multiple different OpStrategy for an op.

This would help support a more complicated op and unblocks compile mode
FSDP

[ghstack-poisoned]
@wanchaol
Update on "[dtensor] add StrategyType and TupleStrategy"
2cb743e 
This PR refactors the current StrategyList. It introduces a
StrategyType, which is the base class of Strategy, and it have
two sub strategies:

1. Refactor the previous StrategyList to OpStrategy
2. Add TupleStrategy, the new strategy added to deal with tuple cases where
it could return multiple different OpStrategy for an op.

This would help support a more complicated op and unblocks compile mode
FSDP

[ghstack-poisoned]
@wanchaol
Update on "[dtensor] add StrategyType and TupleStrategy"
6acb6b6 
This PR refactors the current StrategyList. It introduces a
StrategyType, which is the base class of Strategy, and it have
two sub strategies:

1. Refactor the previous StrategyList to OpStrategy
2. Add TupleStrategy, the new strategy added to deal with tuple cases where
it could return multiple different OpStrategy for an op.

This would help support a more complicated op and unblocks compile mode
FSDP

[ghstack-poisoned]
@wanchaol
Update on "[dtensor] add StrategyType and TupleStrategy"
f07da91 
This PR refactors the current StrategyList. It introduces a
StrategyType, which is the base class of Strategy, and it have
two sub strategies:

1. Refactor the previous StrategyList to OpStrategy
2. Add TupleStrategy, the new strategy added to deal with tuple cases where
it could return multiple different OpStrategy for an op.

This would help support a more complicated op and unblocks compile mode
FSDP

[ghstack-poisoned]
@facebook-github-bot facebook-github-bot deleted the gh/wanchaol/294/head branch June 8, 2023 19:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mrshenli mrshenli mrshenli approved these changes

@zhaojuanmao zhaojuanmao Awaiting requested review from zhaojuanmao

@rohan-varma rohan-varma Awaiting requested review from rohan-varma

@H-Huang H-Huang Awaiting requested review from H-Huang

@awgu awgu Awaiting requested review from awgu

@kwen2501 kwen2501 Awaiting requested review from kwen2501

@fegin fegin Awaiting requested review from fegin

@kiukchung kiukchung Awaiting requested review from kiukchung

@d4l3k d4l3k Awaiting requested review from d4l3k

Assignees

No one assigned

Labels

Merged release notes: distributed (dtensor) release notes category

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@wanchaol @mrshenli @pytorchmergebot