Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Z3: optimizations for grad norm calculation and gradient clipping #5504

Merged
merged 25 commits into from
Aug 15, 2024
Merged

Z3: optimizations for grad norm calculation and gradient clipping #5504

merged 25 commits into from
Aug 15, 2024

Commits on May 9, 2024

  1. z3 scaled_global_grad_norm: repalce get_global_norm with torch.norm

    @nelyahu
    nelyahu committed May 9, 2024
    Configuration menu
    Copy the full SHA
    43792bb View commit details
    Browse the repository at this point in the history

Commits on May 20, 2024

  1. Merge branch 'master' into zero3_scaled_global

    @tjruwase
    tjruwase committed May 20, 2024
    Configuration menu
    Copy the full SHA
    cac04d9 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into zero3_scaled_global

    @loadams
    loadams committed May 20, 2024
    Configuration menu
    Copy the full SHA
    cbb6b6a View commit details
    Browse the repository at this point in the history

Commits on May 23, 2024

  1. Merge branch 'master' into zero3_scaled_global

    @tjruwase
    tjruwase committed May 23, 2024
    Configuration menu
    Copy the full SHA
    37b4cb7 View commit details
    Browse the repository at this point in the history

Commits on May 27, 2024

  1. fix grad norm calc in cpu offload and use torch.clip for grad clipping

    @nelyahu
    nelyahu committed May 27, 2024
    Configuration menu
    Copy the full SHA
    5dd50c3 View commit details
    Browse the repository at this point in the history

Commits on May 30, 2024

  1. Merge branch 'master' into zero3_scaled_global

    @lekurile
    lekurile committed May 30, 2024
    Configuration menu
    Copy the full SHA
    f918054 View commit details
    Browse the repository at this point in the history

Commits on Jun 7, 2024

  1. Merge branch 'master' into zero3_scaled_global

    @nelyahu
    nelyahu committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    ba9fd42 View commit details
    Browse the repository at this point in the history

Commits on Jun 17, 2024

  1. Merge branch 'master' into zero3_scaled_global

    @nelyahu
    nelyahu committed Jun 17, 2024
    Configuration menu
    Copy the full SHA
    238ab34 View commit details
    Browse the repository at this point in the history

Commits on Jun 26, 2024

  1. Merge branch 'master' into zero3_scaled_global

    @loadams
    loadams committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    6b6a834 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into zero3_scaled_global

    @tjruwase
    tjruwase committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    b14f920 View commit details
    Browse the repository at this point in the history

Commits on Jun 27, 2024

  1. Merge branch 'master' into zero3_scaled_global

    @tjruwase
    tjruwase committed Jun 27, 2024
    Configuration menu
    Copy the full SHA
    45e62d2 View commit details
    Browse the repository at this point in the history

Commits on Jul 9, 2024

  1. Merge branch 'master' into zero3_scaled_global

    @loadams
    loadams committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    ffdb7f7 View commit details
    Browse the repository at this point in the history

Commits on Jul 18, 2024

  1. adding gradient clipping to TestZeroPartialOffloadConfigSweep 

    this will allow covering the gradient clipping flow with zero3
and catch issues, such as was observed during this PR
    @nelyahu
    nelyahu committed Jul 18, 2024
    Configuration menu
    Copy the full SHA
    d99524b View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into zero3_scaled_global

    @tjruwase
    tjruwase committed Jul 18, 2024
    Configuration menu
    Copy the full SHA
    e5d5d7c View commit details
    Browse the repository at this point in the history

Commits on Jul 23, 2024

  1. Merge branch 'master' into zero3_scaled_global

    @loadams
    loadams committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    67d2e35 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into zero3_scaled_global

    @loadams
    loadams committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    214103d View commit details
    Browse the repository at this point in the history

  3. Merge branch 'master' into zero3_scaled_global

    @loadams
    loadams committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    6134da4 View commit details
    Browse the repository at this point in the history

Commits on Jul 24, 2024

  1. Merge branch 'master' into zero3_scaled_global

    @loadams
    loadams committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    4b16d7c View commit details
    Browse the repository at this point in the history

Commits on Aug 4, 2024

  1. Merge branch 'master' into zero3_scaled_global

    @tjruwase
    tjruwase committed Aug 4, 2024
    Configuration menu
    Copy the full SHA
    38bb860 View commit details
    Browse the repository at this point in the history

Commits on Aug 6, 2024

  1. Merge branch 'master' into zero3_scaled_global

    @loadams
    loadams committed Aug 6, 2024
    Configuration menu
    Copy the full SHA
    ec1aa8c View commit details
    Browse the repository at this point in the history

Commits on Aug 9, 2024

  1. Merge branch 'master' into zero3_scaled_global

    @loadams
    loadams committed Aug 9, 2024
    Configuration menu
    Copy the full SHA
    95cd3a1 View commit details
    Browse the repository at this point in the history

Commits on Aug 14, 2024

  1. Merge branch 'master' into zero3_scaled_global

    @BacharL
    BacharL committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    26676f2 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into zero3_scaled_global

    @loadams
    loadams committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    e455962 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'master' into zero3_scaled_global

    @loadams
    loadams committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    3407e93 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'master' into zero3_scaled_global

    @loadams
    loadams committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    cb51597 View commit details
    Browse the repository at this point in the history