Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[refactor] move process group from _DistSpec to ColoTensor. #1203

Merged
merged 17 commits into from
Jul 6, 2022
Merged

[refactor] move process group from _DistSpec to ColoTensor. #1203

merged 17 commits into from
Jul 6, 2022

Conversation

feifeibear
Copy link
Contributor

@feifeibear feifeibear commented Jul 5, 2022 •
edited
Loading

process group of a ColoTensor never change.
We should avoid setting dist spec with a process group as a parameter.

Some significant changes.

  1. move process group to ColoTensor
  2. colo context init initialize ColoTensor using a DP ProcessGroup
  3. allow process group change only if pg is DP-only and dist spec is replicate.

@feifeibear feifeibear marked this pull request as draft July 5, 2022 10:18
@feifeibear feifeibear changed the title Tensor/pg4 [refactor] move process group from _DistSpec to ColoTensor. Jul 5, 2022
@feifeibear feifeibear marked this pull request as ready for review July 6, 2022 06:16
@feifeibear feifeibear requested a review from ver217 July 6, 2022 06:18
ver217
ver217 reviewed Jul 6, 2022
View reviewed changes
colossalai/nn/graph/utils.py Outdated Show resolved Hide resolved
@feifeibear
Copy link
Contributor Author

image

@feifeibear
Copy link
Contributor Author

image

@feifeibear feifeibear merged commit ae7d3f4 into hpcaitech:main Jul 6, 2022
@feifeibear feifeibear deleted the tensor/pg4 branch July 6, 2022 08:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ver217 ver217 ver217 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@feifeibear @ver217