[FSDP] Fix pin_memory() for CPU offloading

[awgu]

Stack from ghstack:

• [Easy][FSDP] Change assert -> p_assert #85052 [Easy][FSDP] Change assert -> p_assert
• [Easy][FSDP] Remove outdated comment #85051 [Easy][FSDP] Remove outdated comment
• [FSDP] Fix pin_memory() for CPU offloading #85048 [FSDP] Fix pin_memory() for CPU offloading
• [FSDP] Add rate limiter #83917 [FSDP] Add rate limiter
• [FSDP][Easy] Save unpadded/padded unsharded sizes as attributes #84366 [FSDP][Easy] Save unpadded/padded unsharded sizes as attributes
• [FSDP] Generalize prefetching; lower unshard/reshard to handle #83665 [FSDP] Generalize prefetching; lower unshard/reshard to handle
• [FSDP][Easy] Minor cleanup #84761 [FSDP][Easy] Minor cleanup
• [FSDP] Subtest prefetching for test_fsdp_grad_acc.py #84601 [FSDP] Subtest prefetching for test_fsdp_grad_acc.py
• [FSDP] Remove forward_prefetch #84600 [FSDP] Remove forward_prefetch

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/85048

• 📄 Preview Python docs built from this PR
• 📄 Preview C++ docs built from this PR
• ❓ Need help or want to give feedback on the CI? Visit our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures, 1 Pending

As of commit 61a5f2c:
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[awgu]

@pytorchbot merge -g

[pytorchmergebot]

@pytorchbot successfully started a merge job. Check the current status here.
The merge job was triggered with the green (-g) flag. This means that your change will be merged once all checks on your PR have passed (ETA: 0-4 Hours). If this is not the intended behavior, feel free to use some of the other merge options in the wiki.
Please reach out to the PyTorch DevX Team with feedback or questions!