[simple_fsdp] Port fx pass to bucket reduce_scatters #157780
Conversation
[ghstack-poisoned]
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/157780
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit 574b069 with merge base 02a9d90 ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
pytorch-bot
bot
added
ciflow/inductor
module: inductor
oncall: distributed
| orig_wait_nodes, | ||
| orig_wait_node_recursive_users, | ||
| ) = bucket_id_to_bucketed_op_info[bucket_id] | ||
| # unsharded_grads = [ |
There was a problem hiding this comment.
remove this? (or, why is it commented..)
| device = unsharded_grads[0].meta["val"].device | ||
| rank = device.index | ||
| # TODO: need more work if we want to support non-dim-0 sharding (e.g. search for `shard_dim` in FSDP2 codebase) | ||
| shard_dim = 0 |
There was a problem hiding this comment.
can this be asserted actually? (just for safety sake) hmm, i guess the user does not specify the dim in any way in simple fsdp, so we are literally just choosing it here. thats fine i guess.
Porting fx passes for reduce_scatters bucketing (similar to all_gather bucketing) for simple_fsdp and autoparallel testing. cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx ipiszy chenyang78 kadeng muchulee8 amjames chauhang aakhundov [ghstack-poisoned]
| (FileCheck().check("all_gather_into_tensor_out").run(code)) | ||
| ( | ||
| FileCheck() | ||
| .check_count( |
There was a problem hiding this comment.
just out of curiosity, any reason you used 3 separate FileCheck instances instead of just doing it all in one?
There was a problem hiding this comment.
Oh, thanks, it can be done as a one check :)
Porting fx passes for reduce_scatters bucketing (similar to all_gather bucketing) for simple_fsdp and autoparallel testing. cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx ipiszy chenyang78 kadeng muchulee8 amjames chauhang aakhundov [ghstack-poisoned]
|
@pytorchbot merge |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
3 participants
Stack from ghstack (oldest at bottom):
Porting fx passes for reduce_scatters bucketing (similar to all_gather bucketing) for simple_fsdp and autoparallel testing.
cc @H-Huang @awgu @wanchaol @fegin @fduwjj @wz337 @wconstab @d4l3k @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @chenyang78 @kadeng @muchulee8 @amjames @chauhang @aakhundov