[pipelining] Add _PipelineStage runtime #125729
Conversation
[ghstack-poisoned]
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/125729
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit db46937 with merge base 946b96f ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
pytorch-bot
bot
added
oncall: distributed
ghstack-source-id: 4a462a8623af8e4a20fc36ed53d4acaabffe74ad Pull Request resolved: #125729
cc mrshenli pritamdamania87 zhaojuanmao satgera gqchen aazzolini osalpekar jiayisuse H-Huang awgu penguinwu fegin XilunWu wanchaol fduwjj wz337 tianyu-l wconstab yf225 chauhang d4l3k [ghstack-poisoned]
ghstack-source-id: cd9fb47480cc752361de28ced0f0d66ca87fec3d Pull Request resolved: #125729
There was a problem hiding this comment.
Any reason ManualPipelineStage isn't in the same file? It tripped me up a few times that I couldn't figure out what file it was in, and it seems logical to keep all in the same place.
Any particular code to be reviewed carefully? I think most of the changes are reviewed on the branch already so this is mostly code movement but lmk if there are important changes.
|
The original plan was for tracer's stage and manual's stage be in different files (files are more 1:1 mapped with classes back then), and the base be on the manual side. But as consolidation goes on, the base becomes on the tracer side (bc its implementation is more general). I guess it wouldn't be a big deal to merge Re 2nd q: no substantial code change. @wconstab |
cc mrshenli pritamdamania87 zhaojuanmao satgera gqchen aazzolini osalpekar jiayisuse H-Huang awgu penguinwu fegin XilunWu wanchaol fduwjj wz337 tianyu-l wconstab yf225 chauhang d4l3k [ghstack-poisoned]
|
@pytorchbot merge |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
1. Add pipeline schedules: - GPipe - 1F1B - Interleaved 1F1B - LoopedBFS 2. Add basic forward and backward tests: test_schedule.py Pull Request resolved: #125975 Approved by: https://github.com/wconstab ghstack dependencies: #125729
Pull Request resolved: pytorch#125729 Approved by: https://github.com/wconstab
1. Add pipeline schedules: - GPipe - 1F1B - Interleaved 1F1B - LoopedBFS 2. Add basic forward and backward tests: test_schedule.py Pull Request resolved: pytorch#125975 Approved by: https://github.com/wconstab ghstack dependencies: pytorch#125729
Resolves pytorch/PiPPy#1062. Also added a gradient equivalence test. Pull Request resolved: #126114 Approved by: https://github.com/H-Huang ghstack dependencies: #125729, #125975
Stack from ghstack (oldest at bottom):
cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @awgu @penguinwu @fegin @XilunWu @wanchaol @fduwjj @wz337 @tianyu-l @wconstab @yf225 @chauhang @d4l3k