feature: add support for SMDDP collectives in PytorchDDP distribution #3490

vishwakaria · 2022-11-28T19:41:48Z

Description of changes:
Add support for SMDDP collectives in PT DDP distribution via a new parameter in the dictionary:

communication_options: {
                        "backend": "nccl",  #default value is auto
                       }

Testing done:

Added unit tests
Created a custom docker image and ran successful training

Related Training Toolkit PR: aws/sagemaker-training-toolkit#162

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

I have read the CONTRIBUTING doc
I certify that the changes I am introducing will be backward compatible, and I have discussed concerns about this, if any, with the Python SDK team
I used the commit message format described in CONTRIBUTING
[] I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
[ TBD ] I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
I have checked that my tests are not configured for a specific region or account (if appropriate)
I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

sagemaker-bot · 2022-11-28T20:17:50Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: aa37b66
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-11-28T20:19:34Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: aa37b66
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-11-28T20:26:19Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: aa37b66
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-11-28T20:36:53Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: aa37b66
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

codecov-commenter · 2022-11-28T20:37:05Z

Codecov Report

Merging #3490 (87579b8) into master (1fa2377) will decrease coverage by 0.39%.
The diff coverage is 87.96%.

@@            Coverage Diff             @@
##           master    #3490      +/-   ##
==========================================
- Coverage   89.17%   88.79%   -0.39%     
==========================================
  Files         204      227      +23     
  Lines       18979    22186    +3207     
==========================================
+ Hits        16924    19699    +2775     
- Misses       2055     2487     +432

Impacted Files	Coverage Δ
src/sagemaker/dataset_definition/inputs.py	`100.00% <ø> (ø)`
src/sagemaker/huggingface/model.py	`89.87% <ø> (ø)`
src/sagemaker/lineage/_utils.py	`100.00% <ø> (+12.50%)`	⬆️
src/sagemaker/pipeline.py	`85.33% <ø> (ø)`
src/sagemaker/pytorch/estimator.py	`87.03% <ø> (-12.97%)`	⬇️
src/sagemaker/pytorch/training_compiler/config.py	`84.61% <ø> (ø)`
src/sagemaker/rl/estimator.py	`93.05% <ø> (ø)`
src/sagemaker/session.py	`74.09% <ø> (+2.42%)`	⬆️
src/sagemaker/session_settings.py	`100.00% <ø> (ø)`
src/sagemaker/sklearn/estimator.py	`93.93% <ø> (-6.07%)`	⬇️
... and 99 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

sagemaker-bot · 2022-11-28T20:49:33Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: aa37b66
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-11-29T19:50:59Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: b898588
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-11-29T19:59:28Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: b898588
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-11-29T20:22:24Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: b898588
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-11-29T20:35:09Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: b898588
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-11-29T20:48:07Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: b898588
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

vishwakaria · 2022-11-30T20:04:33Z

All test failures seem to be unrelated:

sagemaker-python-sdk-notebook-tests:
FAILED ./amazon-sagemaker-examples/sagemaker-python-sdk/1P_kmeans_lowlevel/kmeans_mnist_lowlevel.ipynb PT10M14.869S

sagemaker-python-sdk-pr:
FAILED tests/integ/test_profiler.py::test_mxnet_with_custom_profiler_config_then_update_rule_and_config[1.9.0-py38]
FAILED tests/integ/test_profiler.py::test_mxnet_with_default_profiler_config_and_profiler_rule[1.9.0-py38]
FAILED tests/integ/test_profiler.py::test_mxnet_with_profiler_and_debugger_then_disable_framework_metrics[1.9.0-py38]
FAILED tests/integ/test_profiler.py::test_mxnet_with_built_in_profiler_rule_with_custom_parameters[1.9.0-py38]

sagemaker-python-sdk-slow-tests:
FAILED tests/integ/test_inference_pipeline.py::test_inference_pipeline_model_deploy_and_update_endpoint
FAILED tests/integ/test_clarify_model_monitor.py::test_run_explainability_monitor

sagemaker-bot · 2022-11-30T20:30:31Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: b898588
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-11-30T21:01:31Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: b898588
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-11-30T21:09:26Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: b898588
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-11-30T21:20:48Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: b898588
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-11-30T21:30:22Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: b898588
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-12-01T01:52:44Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 8ee0040
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-12-01T02:23:45Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 8ee0040
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-12-01T02:28:41Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 8ee0040
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-12-01T02:30:07Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 8ee0040
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-12-01T02:53:07Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 8ee0040
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

navinsoni

/bot run slow-tests

sagemaker-bot · 2022-12-12T21:14:44Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 558e3f8
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-12-12T21:46:10Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 558e3f8
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-12-12T21:48:19Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 558e3f8
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-12-12T22:04:47Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 558e3f8
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2022-12-12T22:14:22Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 558e3f8
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

* Set to true by default * Update pt ddp supported versions Add validation for instance type, fw and py versions. Clean up logs and comments Add integ test for ACCL disabled Fix integ test params Fix formatting errors in tests Update logic to set accl_enabled to none for default on cases

sagemaker-bot · 2023-01-20T08:28:20Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 87579b8
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-01-20T08:30:12Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 87579b8
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-01-20T08:54:33Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 87579b8
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-01-20T09:38:35Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 87579b8
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-01-20T10:32:22Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 86f8155
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-01-20T10:36:45Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 86f8155
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-01-20T10:38:51Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 86f8155
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-01-20T10:53:41Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 86f8155
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-01-20T12:48:14Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 86f8155
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mufaddal-rohawala · 2023-05-09T01:31:40Z

Closing as this has been in draft state for too long! Please reopen if the change is still relevant.

vishwakaria mentioned this pull request Nov 28, 2022

[DRAFT: Do not merge] feature: enable SMDDP for Pytorch DDP distribution #3355

Closed

9 tasks

vishwakaria requested a review from navinsoni November 30, 2022 20:04

vishwakaria mentioned this pull request Nov 30, 2022

feature: add support for SMDDP collectives to smdataparallel runner aws/sagemaker-training-toolkit#162

Merged

6 tasks

vishwakaria requested a review from knikure November 30, 2022 23:31

knikure force-pushed the smddp-coll branch from b898588 to 8ee0040 Compare December 1, 2022 01:46

knikure self-assigned this Dec 1, 2022

navinsoni reviewed Dec 2, 2022

View reviewed changes

knikure added the do-not-merge Use when PR needs to be marked as do not merge label Dec 2, 2022

vishwakaria force-pushed the smddp-coll branch from 79b225d to 558e3f8 Compare December 12, 2022 21:06

mufaddal-rohawala force-pushed the master branch from bae3b8b to d000bdc Compare December 20, 2022 00:31

vishwakaria added 8 commits January 20, 2023 07:38

Update API spec to use communication_options

6c1afa5

Remove extra whitespace in logs

890723d

Fix unit tests

d42482c

Update comments

b78b404

Update logs

2df8724

Update warning logs

d8dcbd5

Fix nit

094ee86

vishwakaria force-pushed the smddp-coll branch from 558e3f8 to 094ee86 Compare January 20, 2023 07:41

Update supported fw version to 1.13.1

87579b8

Update fw version in unit tests

86f8155

navinsoni marked this pull request as draft March 8, 2023 20:42

mufaddal-rohawala closed this May 9, 2023

feature: add support for SMDDP collectives in PytorchDDP distribution #3490

feature: add support for SMDDP collectives in PytorchDDP distribution #3490

Uh oh!

Conversation

vishwakaria commented Nov 28, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge Checklist

General

Tests

Uh oh!

sagemaker-bot commented Nov 28, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Nov 28, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Nov 28, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Nov 28, 2022

AWS CodeBuild CI Report

Uh oh!

codecov-commenter commented Nov 28, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

sagemaker-bot commented Nov 28, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Nov 29, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Nov 29, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Nov 29, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Nov 29, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Nov 29, 2022

AWS CodeBuild CI Report

Uh oh!

vishwakaria commented Nov 30, 2022

Uh oh!

sagemaker-bot commented Nov 30, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Nov 30, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Nov 30, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Nov 30, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Nov 30, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Dec 1, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Dec 1, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Dec 1, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Dec 1, 2022

AWS CodeBuild CI Report

Uh oh!

sagemaker-bot commented Dec 1, 2022

AWS CodeBuild CI Report

Uh oh!

navinsoni left a comment

Choose a reason for hiding this comment

Uh oh!

sagemaker-bot commented Dec 12, 2022

vishwakaria commented Nov 28, 2022 •

edited

Loading

codecov-commenter commented Nov 28, 2022 •

edited

Loading