Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FSDPv1] Only perform cat() during last microbatch backward() within FlattenParamsWrapper #1180

Draft
wants to merge 27 commits into
base: ngoyal_changes_for_pp_fp8
Choose a base branch
from
Draft

[FSDPv1] Only perform cat() during last microbatch backward() within FlattenParamsWrapper #1180

wants to merge 27 commits into from

Commits on Apr 29, 2024

  1. use torch.no_grad() to avoid calling cat() during FSDP backward excep… 

    …t for last microbatch
    @chrisxcai
    chrisxcai committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    d1102ce View commit details
    Browse the repository at this point in the history

  2. remove logging

    @chrisxcai
    chrisxcai committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    9a22628 View commit details
    Browse the repository at this point in the history

Commits on Apr 30, 2024

  1. logging

    @chrisxcai
    chrisxcai committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    f787532 View commit details
    Browse the repository at this point in the history

Commits on May 1, 2024

  1. logging

    @chrisxcai
    chrisxcai committed May 1, 2024
    Configuration menu
    Copy the full SHA
    3429f33 View commit details
    Browse the repository at this point in the history

Commits on May 2, 2024

  1. use new field to accumulate per-parameter grads in fp32 and copy into… 

    … flatten_parameter.unsharded_main_grad in last microbatch backward()
    @chrisxcai
    chrisxcai committed May 2, 2024
    Configuration menu
    Copy the full SHA
    4b5abe2 View commit details
    Browse the repository at this point in the history

  2. clean up accumulated fp32 grads between data batches

    @chrisxcai
    chrisxcai committed May 2, 2024
    Configuration menu
    Copy the full SHA
    c97bfd9 View commit details
    Browse the repository at this point in the history

  3. logging

    @chrisxcai
    chrisxcai committed May 2, 2024
    Configuration menu
    Copy the full SHA
    d2a88b7 View commit details
    Browse the repository at this point in the history

Commits on May 6, 2024

  1. logging

    @chrisxcai
    chrisxcai committed May 6, 2024
    Configuration menu
    Copy the full SHA
    901fb86 View commit details
    Browse the repository at this point in the history

Commits on May 8, 2024

  1. return grad in post_backward_hook()

    @chrisxcai
    chrisxcai committed May 8, 2024
    Configuration menu
    Copy the full SHA
    ad40f24 View commit details
    Browse the repository at this point in the history

Commits on May 9, 2024

  1. correct param_index

    @chrisxcai
    chrisxcai committed May 9, 2024
    Configuration menu
    Copy the full SHA
    14499fe View commit details
    Browse the repository at this point in the history

  2. logging

    @chrisxcai
    chrisxcai committed May 9, 2024
    Configuration menu
    Copy the full SHA
    ad7aa1f View commit details
    Browse the repository at this point in the history

  3. add torch.testing.assert_allclose() to compare baseline and new grads

    @chrisxcai
    chrisxcai committed May 9, 2024
    Configuration menu
    Copy the full SHA
    b835770 View commit details
    Browse the repository at this point in the history

  4. logging

    @chrisxcai
    chrisxcai committed May 9, 2024
    Configuration menu
    Copy the full SHA
    d689f38 View commit details
    Browse the repository at this point in the history

Commits on May 13, 2024

  1. logging

    @chrisxcai
    chrisxcai committed May 13, 2024
    Configuration menu
    Copy the full SHA
    e8df583 View commit details
    Browse the repository at this point in the history

Commits on May 15, 2024

  1. honor optimize_backward_concat flag

    @chrisxcai
    chrisxcai committed May 15, 2024
    Configuration menu
    Copy the full SHA
    5926a79 View commit details
    Browse the repository at this point in the history

  2. documentation

    @chrisxcai
    chrisxcai committed May 15, 2024
    Configuration menu
    Copy the full SHA
    5d08aa3 View commit details
    Browse the repository at this point in the history

  3. update documentation

    @chrisxcai
    chrisxcai committed May 15, 2024
    Configuration menu
    Copy the full SHA
    c91cb72 View commit details
    Browse the repository at this point in the history

  4. update documentation

    @chrisxcai
    chrisxcai committed May 15, 2024
    Configuration menu
    Copy the full SHA
    fd3f3fc View commit details
    Browse the repository at this point in the history

  5. use grad instead of grad.data

    @chrisxcai
    chrisxcai committed May 15, 2024
    Configuration menu
    Copy the full SHA
    7678503 View commit details
    Browse the repository at this point in the history

  6. clean up

    @chrisxcai
    chrisxcai committed May 15, 2024
    Configuration menu
    Copy the full SHA
    c55a0d1 View commit details
    Browse the repository at this point in the history

  7. Added reshard hook for frozen params in backward

    @awgu @chrisxcai
    awgu authored and chrisxcai committed May 15, 2024
    Configuration menu
    Copy the full SHA
    688b902 View commit details
    Browse the repository at this point in the history

  8. Avoid calling _free_fp16_param_shard() too early with PR 1159

    @jiecaoyu @chrisxcai
    jiecaoyu authored and chrisxcai committed May 15, 2024
    Configuration menu
    Copy the full SHA
    a3ff5c4 View commit details
    Browse the repository at this point in the history

  9. Added requires_grad check for params_with_grad method (#1171) 

    Co-authored-by: Jie Wang <jiewang@meta.com>
    @whbldhwj @chrisxcai
    2 people authored and chrisxcai committed May 15, 2024
    Configuration menu
    Copy the full SHA
    9d0e41e View commit details
    Browse the repository at this point in the history

  10. Changed to only run reshard hook if all gradients computed (#1166) 

    * Changed to only run reshard hook if all gradients computed

* Fix decreasing it/s with multi-grad hook
    @awgu @chrisxcai
    awgu authored and chrisxcai committed May 15, 2024
    Configuration menu
    Copy the full SHA
    e43a22f View commit details
    Browse the repository at this point in the history

  11. Add cast input argument (#1175) 

    Co-authored-by: Jie Wang <jiewang@meta.com>
    @whbldhwj @chrisxcai
    2 people authored and chrisxcai committed May 15, 2024
    Configuration menu
    Copy the full SHA
    f039a3a View commit details
    Browse the repository at this point in the history

  12. honor optimize_backward_concat flag

    @chrisxcai
    chrisxcai committed May 15, 2024
    Configuration menu
    Copy the full SHA
    5299982 View commit details
    Browse the repository at this point in the history

  13. use grad instead of grad.data

    @chrisxcai
    chrisxcai committed May 15, 2024
    Configuration menu
    Copy the full SHA
    b5e138f View commit details
    Browse the repository at this point in the history