Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enable TORCH_NCCL_AVOID_RECORD_STREAMS=1 by default #512

Merged
merged 17 commits into from
Jun 5, 2024
Merged

Enable TORCH_NCCL_AVOID_RECORD_STREAMS=1 by default #512

merged 17 commits into from
Jun 5, 2024

Commits on Jun 3, 2024

  1. Add avoid_torch_nccl_record_streams decorator

    @IvanYashchuk
    IvanYashchuk committed Jun 3, 2024
    Configuration menu
    Copy the full SHA
    960f3cc View commit details
    Browse the repository at this point in the history

  2. Use avoid_torch_nccl_record_streams in Thunder-generated functions by… 

    … default
    @IvanYashchuk
    IvanYashchuk committed Jun 3, 2024
    Configuration menu
    Copy the full SHA
    c436656 View commit details
    Browse the repository at this point in the history

  3. Use '0' as default value for unset env var

    @IvanYashchuk
    IvanYashchuk committed Jun 3, 2024
    Configuration menu
    Copy the full SHA
    62d093e View commit details
    Browse the repository at this point in the history

  4. Set TORCH_NCCL_AVOID_RECORD_STREAMS at Thunder import

    @IvanYashchuk
    IvanYashchuk committed Jun 3, 2024
    Configuration menu
    Copy the full SHA
    6b5c101 View commit details
    Browse the repository at this point in the history

  5. Set TORCH_NCCL_AVOID_RECORD_STREAMS before init_process_group in the … 

    …benchmark script
    @IvanYashchuk
    IvanYashchuk committed Jun 3, 2024
    Configuration menu
    Copy the full SHA
    326c8dc View commit details
    Browse the repository at this point in the history

  6. Revert initial changes they don't really work

    @IvanYashchuk
    IvanYashchuk committed Jun 3, 2024
    Configuration menu
    Copy the full SHA
    7772930 View commit details
    Browse the repository at this point in the history

  7. Instead of default process group create a copy with magic env var set

    @IvanYashchuk
    IvanYashchuk committed Jun 3, 2024
    Configuration menu
    Copy the full SHA
    d2d06cd View commit details
    Browse the repository at this point in the history

  8. Fix partially initialized module error

    @IvanYashchuk
    IvanYashchuk committed Jun 3, 2024
    Configuration menu
    Copy the full SHA
    22205dd View commit details
    Browse the repository at this point in the history

  9. Create tdist.ProcessGroupNCCL.Options

    @IvanYashchuk
    IvanYashchuk committed Jun 3, 2024
    Configuration menu
    Copy the full SHA
    7ece68c View commit details
    Browse the repository at this point in the history

  10. Merge branch 'fix-420-attempt2' into fix-420

    @IvanYashchuk
    IvanYashchuk committed Jun 3, 2024
    Configuration menu
    Copy the full SHA
    695be3d View commit details
    Browse the repository at this point in the history

  11. Revert "Set TORCH_NCCL_AVOID_RECORD_STREAMS at Thunder import" 

    This reverts commit 6b5c101.
    @IvanYashchuk
    IvanYashchuk committed Jun 3, 2024
    Configuration menu
    Copy the full SHA
    9da0288 View commit details
    Browse the repository at this point in the history

Commits on Jun 4, 2024

  1. Fix AttributeError: module 'torch.distributed' has no attribute 'Proc… 

    …essGroupNCCL'
    @IvanYashchuk
    IvanYashchuk committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    0749eda View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into fix-420

    @IvanYashchuk
    IvanYashchuk committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    ff4539f View commit details
    Browse the repository at this point in the history

Commits on Jun 5, 2024

  1. Merge remote-tracking branch 'upstream/main' into fix-420

    @IvanYashchuk
    IvanYashchuk committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    6309dd9 View commit details
    Browse the repository at this point in the history

  2. Fix coalescing manager error

    @IvanYashchuk
    IvanYashchuk committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    8a1deb7 View commit details
    Browse the repository at this point in the history

  3. Fix another coalescing manager error

    @IvanYashchuk
    IvanYashchuk committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    a09b68a View commit details
    Browse the repository at this point in the history

  4. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    fb8d4ac View commit details
    Browse the repository at this point in the history