[Replicate][Test] tests that pp model grads are the same as single-device model grads #164890
Conversation
…vice model grads [ghstack-poisoned]
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/164890
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit 47cac6d with merge base f63d16c ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
pytorch-bot
bot
added
oncall: distributed
test/distributed/_composable/test_composability/test_pp_composability.py
Outdated
Show resolved
Hide resolved
test/distributed/_composable/test_composability/test_pp_composability.py
Show resolved
Hide resolved
| ], | ||
| ) | ||
| def test_replicate_pp_grads(self, ScheduleClass): | ||
| _device_raii = torch.device(device_type, self.device) |
There was a problem hiding this comment.
confused how this is used? My guess based on the variable name is keep a reference to the device to keep it alive, but not sure why it is needed. If you can find out, i am curious!
There was a problem hiding this comment.
I believe its intended purpose was to do what you're saying. Turns out that its not needed in this instance so I will get rid of them.
| pipeline_model_parameter_dict = {} | ||
|
|
||
| # Attach to a schedule | ||
| if issubclass(ScheduleClass, PipelineScheduleSingle): |
There was a problem hiding this comment.
dang, this if-else needed for single stage schedules vs multistage schedules is so ugly, i should fix it 😭
test/distributed/_composable/test_composability/test_pp_composability.py
Outdated
Show resolved
Hide resolved
…s single-device model grads" **Summary:** Created a test so that we can verify that a model that has been pipelined + replicated has the same gradients as a reference model. To do this, I mapped the layers and their parameters in each partial model to the original full model and then compared the gradients. **Test Case** 1. pytest test/distributed/_composable/test_composability/test_pp_composability.py -k test_replicate_pp_grads cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k pragupta msaroufim dcci [ghstack-poisoned]
|
@pytorchbot merge |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
Summary: Created a test so that we can verify that a model that has been pipelined + replicated has the same gradients as a reference model. To do this, I mapped the layers and their parameters in each partial model to the original full model and then compared the gradients.
Test Case
Stack from ghstack (oldest at bottom):
cc @H-Huang @awgu @wanchaol @fegin @fduwjj @wz337 @wconstab @d4l3k @pragupta @msaroufim @dcci