Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adding notebook for MNIST training using PyTorch and StepFunctions #4599

Merged
merged 9 commits into from
Mar 29, 2024
Merged

Adding notebook for MNIST training using PyTorch and StepFunctions #4599

merged 9 commits into from
Mar 29, 2024

Conversation

Avs163
Copy link
Contributor

@Avs163 Avs163 commented Mar 26, 2024

Issue #, if available: Adding notebook and MNIST.py

Description of changes: Adding a notebook to create a simple workflow that involves a training step, creating a model, configuring an endpoint, and deploying the model. The notebook is to run Machine Learning pipelines on Step Functions, via MNIST. The folder originally existed in this same repository but got deleted and hence creating a PR to merge it back.

Testing done:

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

  • I have read the CONTRIBUTING doc and adhered to the example notebook best practices
  • [] I have updated any necessary documentation, including READMEs
  • I have tested my notebook(s) and ensured it runs end-to-end
  • I have linted my notebook(s) and code using black-nb -l 100 {path}/{notebook-name}.ipynb

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@review-notebook-app Review Notebook App
Copy link

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: SageMakerNotebooksProd-USWEST2-amazon-sagemaker-examples-pr
  • Commit ID: e68fdf0
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: SageMakerNotebooksProd-USWEST2-sagemaker-examples-link-check
  • Commit ID: e68fdf0
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: SageMakerNotebooksProd-USWEST2-sagemaker-examples-code-formatting
  • Commit ID: e68fdf0
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: SageMakerNotebooksProd-USWEST2-sagemaker-examples-grammar
  • Commit ID: e68fdf0
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@Avs163
Update mnist.py
c51fcee 
# Set a fixed random seed for reproducibility

SEED = 42
torch.manual_seed(SEED)
np.random.seed(SEED)
random.seed(SEED)
@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: SageMakerNotebooksProd-USWEST2-amazon-sagemaker-examples-pr
  • Commit ID: c51fcee
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: SageMakerNotebooksProd-USWEST2-sagemaker-examples-link-check
  • Commit ID: c51fcee
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: SageMakerNotebooksProd-USWEST2-sagemaker-examples-code-formatting
  • Commit ID: c51fcee
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: SageMakerNotebooksProd-USWEST2-sagemaker-examples-grammar
  • Commit ID: c51fcee
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@Avs163
Update mnist.py
5f6910d 
The main change is replacing with torch.no_grad(): with with torch.inference_mode():.
@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: SageMakerNotebooksProd-USWEST2-amazon-sagemaker-examples-pr
  • Commit ID: 5f6910d
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: SageMakerNotebooksProd-USWEST2-sagemaker-examples-link-check
  • Commit ID: 5f6910d
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: SageMakerNotebooksProd-USWEST2-sagemaker-examples-code-formatting
  • Commit ID: 5f6910d
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: SageMakerNotebooksProd-USWEST2-sagemaker-examples-grammar
  • Commit ID: 5f6910d
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

Avs163
Avs163 commented Mar 27, 2024
View reviewed changes
Copy link
Contributor Author

@Avs163 Avs163 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Made three changes as recommended by codeguru:

  1. Added a random seed for reproducibility
  2. Verified that model.eval() is called before testing loop executes
  3. Replaced torch.no_grad(): with with torch.inference_mode():

step-functions-data-science-sdk/training_pipeline_pytorch_mnist/code/mnist.py Show resolved Hide resolved
step-functions-data-science-sdk/training_pipeline_pytorch_mnist/code/mnist.py Show resolved Hide resolved
step-functions-data-science-sdk/training_pipeline_pytorch_mnist/code/mnist.py Show resolved Hide resolved
Avs163
Avs163 commented Mar 27, 2024
View reviewed changes
Copy link
Contributor Author

@Avs163 Avs163 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replaced torch.no_grad(): with with torch.inference_mode():.

@zhaoqizqwang
Copy link
Contributor

Could you take a look at the examples-pr and code formatting check failures? Meanwhile readthedocs failure is not related and I'm investigating it

@zhaoqizqwang
Copy link
Contributor

example-pr logs says badge is missing. https://github.com/aws/amazon-sagemaker-examples/blob/main/CONTRIBUTING.md Here is the guide to include CI badge

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: SageMakerNotebooksProd-USWEST2-amazon-sagemaker-examples-pr
  • Commit ID: c59ecea
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: SageMakerNotebooksProd-USWEST2-sagemaker-examples-link-check
  • Commit ID: c59ecea
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: SageMakerNotebooksProd-USWEST2-sagemaker-examples-code-formatting
  • Commit ID: c59ecea
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: SageMakerNotebooksProd-USWEST2-sagemaker-examples-grammar
  • Commit ID: c59ecea
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: SageMakerNotebooksProd-USWEST2-sagemaker-examples-code-formatting
  • Commit ID: 1e196d5
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: SageMakerNotebooksProd-USWEST2-sagemaker-examples-link-check
  • Commit ID: 1e196d5
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: SageMakerNotebooksProd-USWEST2-sagemaker-examples-grammar
  • Commit ID: 1e196d5
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: SageMakerNotebooksProd-USWEST2-sagemaker-examples-link-check
  • Commit ID: a9e903b
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: SageMakerNotebooksProd-USWEST2-sagemaker-examples-code-formatting
  • Commit ID: a9e903b
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@Avs163
Copy link
Contributor Author

Avs163 commented Mar 29, 2024

example-pr logs says badge is missing. https://github.com/aws/amazon-sagemaker-examples/blob/main/CONTRIBUTING.md Here is the guide to include CI badge

I have added CI badge in the code. Please let me know if any other changes need to be made

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: SageMakerNotebooksProd-USWEST2-sagemaker-examples-grammar
  • Commit ID: a9e903b
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: SageMakerNotebooksProd-USWEST2-amazon-sagemaker-examples-pr
  • Commit ID: a9e903b
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: SageMakerNotebooksProd-USWEST2-amazon-sagemaker-examples-pr
  • Commit ID: 4dbb330
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: SageMakerNotebooksProd-USWEST2-sagemaker-examples-link-check
  • Commit ID: 4dbb330
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: SageMakerNotebooksProd-USWEST2-sagemaker-examples-code-formatting
  • Commit ID: 4dbb330
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: SageMakerNotebooksProd-USWEST2-sagemaker-examples-grammar
  • Commit ID: 4dbb330
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: SageMakerNotebooksProd-USWEST2-amazon-sagemaker-examples-pr
  • Commit ID: a976752
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: SageMakerNotebooksProd-USWEST2-sagemaker-examples-link-check
  • Commit ID: a976752
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: SageMakerNotebooksProd-USWEST2-sagemaker-examples-code-formatting
  • Commit ID: a976752
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@sagemaker-bot
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: SageMakerNotebooksProd-USWEST2-sagemaker-examples-grammar
  • Commit ID: a976752
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@Avs163
Copy link
Contributor Author

Avs163 commented Mar 29, 2024

Reformatted the code, and now all required checks are passed.

zhaoqizqwang
zhaoqizqwang approved these changes Mar 29, 2024
View reviewed changes
Copy link
Contributor

@zhaoqizqwang zhaoqizqwang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Skipping grammar check

@zhaoqizqwang zhaoqizqwang merged commit 7f1133e into aws:main Mar 29, 2024
4 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sagemaker-bot sagemaker-bot sagemaker-bot left review comments

@zhaoqizqwang zhaoqizqwang zhaoqizqwang approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@Avs163 @sagemaker-bot @zhaoqizqwang