Try PyTorch FSDP "HYBRID_SHARD" strategy #38
Labels
difficulty/easy
Easy issue to tackle, may take a couple hours
project/model
Related to modeling decisions and implementations
severity/should
Something that should be implemented/fixed
status/blocked
Progress can't be made because we're waiting on something outside of our control
While this wasn't implemented as of PyTorch 1.13.1, it appears it's going to be in the next release because it's implemented in the master branch: https://github.com/pytorch/pytorch/blob/master/torch/distributed/fsdp/api.py#L31.
The text was updated successfully, but these errors were encountered: