Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add async grad allreduce and chunk optimization #4084

Merged
merged 28 commits into from
May 17, 2022
Merged

add async grad allreduce and chunk optimization #4084

merged 28 commits into from
May 17, 2022

Commits on Mar 18, 2022

  1. O2 runs but O1 does not 

    Signed-off-by: ericharper <complex451@gmail.com>
    @ericharper
    ericharper committed Mar 18, 2022
    Configuration menu
    Copy the full SHA
    b3f7a84 View commit details
    Browse the repository at this point in the history

Commits on Mar 22, 2022

  1. disable async for O1 

    Signed-off-by: ericharper <complex451@gmail.com>
    @ericharper
    ericharper committed Mar 22, 2022
    Configuration menu
    Copy the full SHA
    b91f0e3 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into gpt_sync_handler

    @ericharper
    ericharper committed Mar 22, 2022
    Configuration menu
    Copy the full SHA
    32120c9 View commit details
    Browse the repository at this point in the history

Commits on Mar 23, 2022

  1. typo 

    Signed-off-by: ericharper <complex451@gmail.com>
    @ericharper
    ericharper committed Mar 23, 2022
    Configuration menu
    Copy the full SHA
    f49ec58 View commit details
    Browse the repository at this point in the history

  2. update async flag in configure_optimizers 

    Signed-off-by: ericharper <complex451@gmail.com>
    @ericharper
    ericharper committed Mar 23, 2022
    Configuration menu
    Copy the full SHA
    73eb1a5 View commit details
    Browse the repository at this point in the history

  3. typo 

    Signed-off-by: ericharper <complex451@gmail.com>
    @ericharper
    ericharper committed Mar 23, 2022
    Configuration menu
    Copy the full SHA
    d79cee2 View commit details
    Browse the repository at this point in the history

Commits on Mar 24, 2022

  1. Merge branch 'main' into gpt_sync_handler

    @ericharper
    ericharper committed Mar 24, 2022
    Configuration menu
    Copy the full SHA
    f9355ee View commit details
    Browse the repository at this point in the history

  2. revert 

    Signed-off-by: ericharper <complex451@gmail.com>
    @ericharper
    ericharper committed Mar 24, 2022
    Configuration menu
    Copy the full SHA
    8fef3a9 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'gpt_sync_handler' of github.com:NVIDIA/NeMo into gpt_sy… 

    …nc_handler
    @ericharper
    ericharper committed Mar 24, 2022
    Configuration menu
    Copy the full SHA
    2e4da67 View commit details
    Browse the repository at this point in the history

  4. update _require if using async 

    Signed-off-by: ericharper <complex451@gmail.com>
    @ericharper
    ericharper committed Mar 24, 2022
    Configuration menu
    Copy the full SHA
    5f7a45b View commit details
    Browse the repository at this point in the history

Commits on Mar 25, 2022

  1. Merge branch 'main' into gpt_sync_handler

    @ericharper
    ericharper committed Mar 25, 2022
    Configuration menu
    Copy the full SHA
    21e8e34 View commit details
    Browse the repository at this point in the history

Commits on Apr 18, 2022

  1. Merge branch 'main' into gpt_sync_handler

    @ericharper
    ericharper committed Apr 18, 2022
    Configuration menu
    Copy the full SHA
    eabaf62 View commit details
    Browse the repository at this point in the history

Commits on Apr 21, 2022

  1. Merge branch 'main' into gpt_sync_handler

    @ericharper
    ericharper committed Apr 21, 2022
    Configuration menu
    Copy the full SHA
    e0271a8 View commit details
    Browse the repository at this point in the history

  2. clean comments 

    Signed-off-by: ericharper <complex451@gmail.com>
    @ericharper
    ericharper committed Apr 21, 2022
    Configuration menu
    Copy the full SHA
    fcb58de View commit details
    Browse the repository at this point in the history

Commits on Apr 27, 2022

  1. always all_reduce 

    Signed-off-by: ericharper <complex451@gmail.com>
    @ericharper
    ericharper committed Apr 27, 2022
    Configuration menu
    Copy the full SHA
    4761b24 View commit details
    Browse the repository at this point in the history

Commits on Apr 28, 2022

  1. add async grad allreduce and chunk optimization to T5

    @xrennvidia
    xrennvidia committed Apr 28, 2022
    Configuration menu
    Copy the full SHA
    b0ba0a9 View commit details
    Browse the repository at this point in the history

Commits on Apr 29, 2022

  1. Merge branch 'main' into xren/t5_sync_handler

    @ericharper
    ericharper committed Apr 29, 2022
    Configuration menu
    Copy the full SHA
    08b2a70 View commit details
    Browse the repository at this point in the history

  2. push reformatted files after style check

    @xrennvidia
    xrennvidia committed Apr 29, 2022
    Configuration menu
    Copy the full SHA
    04eac5e View commit details
    Browse the repository at this point in the history

  3. Merge branch 'main' into xren/t5_sync_handler

    @ericharper
    ericharper committed Apr 29, 2022
    Configuration menu
    Copy the full SHA
    db9cbb9 View commit details
    Browse the repository at this point in the history

Commits on Apr 30, 2022

  1. set chunk size as 0 while async grad allreduce is off

    @xrennvidia
    xrennvidia committed Apr 30, 2022
    Configuration menu
    Copy the full SHA
    28b795c View commit details
    Browse the repository at this point in the history

Commits on May 3, 2022

  1. more experiments show that 125MB is a better default chunk size for m… 

    …ost cases
    @xrennvidia
    xrennvidia committed May 3, 2022
    Configuration menu
    Copy the full SHA
    f6b2f3b View commit details
    Browse the repository at this point in the history

  2. add grad_allreduce_chunk_size_mb for GPT-3

    @xrennvidia
    xrennvidia committed May 3, 2022
    Configuration menu
    Copy the full SHA
    382c533 View commit details
    Browse the repository at this point in the history

Commits on May 11, 2022

  1. at the end of each training step, wait until all async grad allreduce… 

    … works are done
    @xrennvidia
    xrennvidia committed May 11, 2022
    Configuration menu
    Copy the full SHA
    36dcfd2 View commit details
    Browse the repository at this point in the history

  2. replace individual allreduce work.wait() with a single dGPU evice syn… 

    …chroonization
    @xrennvidia
    xrennvidia committed May 11, 2022
    Configuration menu
    Copy the full SHA
    e7e8124 View commit details
    Browse the repository at this point in the history

Commits on May 12, 2022

  1. record the status of each allreduce work seems too much for perf

    @xrennvidia
    xrennvidia committed May 12, 2022
    Configuration menu
    Copy the full SHA
    6b8b59e View commit details
    Browse the repository at this point in the history

  2. add more comments

    @xrennvidia
    xrennvidia committed May 12, 2022
    Configuration menu
    Copy the full SHA
    2280634 View commit details
    Browse the repository at this point in the history

  3. push a reformatted file

    @xrennvidia
    xrennvidia committed May 12, 2022
    Configuration menu
    Copy the full SHA
    0469315 View commit details
    Browse the repository at this point in the history

Commits on May 17, 2022

  1. Merge branch 'main' into xren/t5_sync_handler

    @ericharper
    ericharper committed May 17, 2022
    Configuration menu
    Copy the full SHA
    2ba62cf View commit details
    Browse the repository at this point in the history