Skip to content

Commit

Permalink
Update on "[FSDP] Clarify CPU offload implicitly in reshard_doc"
Browse files Browse the repository at this point in the history
Per title

Differential Revision: [D44812344](https://our.internmc.facebook.com/intern/diff/D44812344/)

[ghstack-poisoned]
  • Loading branch information
rohan-varma committed Apr 10, 2023
1 parent dcf5662 commit 5abb754
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion torch/distributed/fsdp/flat_param.py
Original file line number Diff line number Diff line change
Expand Up @@ -1489,7 +1489,7 @@ def reshard(self, free_unsharded_flat_param: bool):
parameter if ``free_unsharded_flat_param`` and switching to using the
sharded flat parameter. Note that this also implicitly offloads
the sharded flat parameter (if CPU offload is enabled) by pointing
it to the local_shard attribute which resides on CPU.
it to the ``_local_shard`` attribute which resides on CPU.
"""
# Switch to the sharded `FlatParameter` before freeing to prevent
# "use-after-free"-type bugs with external profiling tools, where for
Expand Down

0 comments on commit 5abb754

Please sign in to comment.