feat: Add transformers 4.29 - PyTorch 2.0.0 training and inference #3904

phall1 · 2023-06-06T02:01:17Z

Issue #, if available: #3896

Description of changes:
Update Image URI for:

Transformers 4.29 - PyTorch 2.0 Inference DLC
Transformers 4.29 - PyTorch 2.0 Training DLC

Testing done:

Merge Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your pull request.

General

I have read the CONTRIBUTING doc
I certify that the changes I am introducing will be backward compatible, and I have discussed concerns about this, if any, with the Python SDK team
I used the commit message format described in CONTRIBUTING
I have passed the region in to all S3 and STS clients that I've initialized as part of this change.
I have updated any necessary documentation, including READMEs and API docs (if appropriate)

Tests

I have added tests that prove my fix is effective or that my feature works (if appropriate)
I have added unit and/or integration tests as appropriate to ensure backward compatibility of the changes
I have checked that my tests are not configured for a specific region or account (if appropriate)
I have used unique_name_from_base to create resource names in integ tests (if appropriate)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

knikure

/bot run all

sagemaker-bot · 2023-06-06T06:03:39Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 9be32b3
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-06-06T06:04:42Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 9be32b3
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-06-06T06:04:47Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 9be32b3
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-06-06T06:42:27Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 9be32b3
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-06-06T07:17:35Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 9be32b3
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

codecov-commenter · 2023-06-06T07:19:29Z

Codecov Report

Attention: 32 lines in your changes are missing coverage. Please review.

Comparison is base (3057f21) 90.27% compared to head (a494586) 90.23%.
Report is 64 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3904      +/-   ##
==========================================
- Coverage   90.27%   90.23%   -0.04%     
==========================================
  Files         993      316     -677     
  Lines       87861    29282   -58579     
==========================================
- Hits        79313    26424   -52889     
+ Misses       8548     2858    -5690

Files	Coverage Δ
src/sagemaker/base_predictor.py	`90.90% <100.00%> (ø)`
src/sagemaker/config/config_schema.py	`100.00% <100.00%> (ø)`
src/sagemaker/estimator.py	`92.88% <ø> (ø)`
src/sagemaker/feature_store/feature_group.py	`78.82% <ø> (ø)`
...emaker/feature_store/feature_processor/__init__.py	`100.00% <ø> (ø)`
...eature_store/feature_processor/_config_uploader.py	`97.43% <100.00%> (ø)`
...er/feature_store/feature_processor/_data_source.py	`100.00% <100.00%> (ø)`
...ore/feature_processor/_feature_processor_config.py	`100.00% <100.00%> (ø)`
.../feature_store/feature_processor/_params_loader.py	`100.00% <ø> (ø)`
...ker/feature_store/feature_processor/_validation.py	`84.05% <100.00%> (ø)`
... and 32 more

... and 1267 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

per the build errors in these tests: aws#3904 (comment) aws#3904 (comment)

phall1 · 2023-06-06T07:58:41Z

Updated python version for torch 2.0 compatibility per the failed tests in the failed builds

per the build errors in these tests: aws#3904 (comment) aws#3904 (comment)

sagemaker-bot · 2023-06-07T05:02:04Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: a84574b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-06-07T05:12:12Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: a84574b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-06-07T05:19:10Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: a84574b
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-06-07T05:20:49Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: a84574b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

phall1 · 2023-06-07T05:27:19Z

Looks like the image isn't available yet, per here:

https://github.com/aws/deep-learning-containers/blob/master/available_images.md

Do we know if this is being worked on or how I could contribute to getting this image available?
Error from logs:

sagemaker.exceptions.UnexpectedStatusException: Error for Processing job test-huggingface-2023-06-07-04-47-23-957: Failed. Reason: ClientError: API error (404): manifest for 763104351884.dkr.ecr.us-west-2.amazonaws.com/huggingface-pytorch-training:2.0.0-transformers4.29.0-gpu-py310-cu118-ubuntu20.04 not found: manifest unknown: Requested image not found

sagemaker-bot · 2023-06-07T05:38:39Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: a84574b
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

mufaddal-rohawala

/bot run all

sagemaker-bot · 2023-06-07T17:05:39Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: aeb5cc3
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-06-07T17:43:26Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: aeb5cc3
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-06-07T17:48:01Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: aeb5cc3
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

knikure

/bot run all

sagemaker-bot · 2023-06-16T17:12:19Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: 2b1464f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-06-16T17:12:26Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: 2b1464f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-06-16T17:12:32Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: 2b1464f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-06-16T18:35:07Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: 2b1464f
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-06-16T19:20:22Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: 2b1464f
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

phall1 · 2023-06-16T19:22:00Z

Looks like the image isn't available yet, per here:

https://github.com/aws/deep-learning-containers/blob/master/available_images.md

Do we know if this is being worked on or how I could contribute to getting this image available?

Error from logs:
sagemaker.exceptions.UnexpectedStatusException: Error for Processing job test-huggingface-2023-06-07-04-47-23-957: Failed. Reason: ClientError: API error (404): manifest for 763104351884.dkr.ecr.us-west-2.amazonaws.com/huggingface-pytorch-training:2.0.0-transformers4.29.0-gpu-py310-cu118-ubuntu20.04 not found: manifest unknown: Requested image not found

Same error as before - bumping my question l

knikure · 2023-06-16T21:03:42Z

Looks like the image isn't available yet, per here:
aws/deep-learning-containers@master/available_images.md
Do we know if this is being worked on or how I could contribute to getting this image available?
Error from logs:
sagemaker.exceptions.UnexpectedStatusException: Error for Processing job test-huggingface-2023-06-07-04-47-23-957: Failed. Reason: ClientError: API error (404): manifest for 763104351884.dkr.ecr.us-west-2.amazonaws.com/huggingface-pytorch-training:2.0.0-transformers4.29.0-gpu-py310-cu118-ubuntu20.04 not found: manifest unknown: Requested image not found
Same error as before - bumping my question l

We will have to check with DLC team to verify on the image availability.
@ShiboXing Can you help answer this question?

knikure · 2023-08-14T21:16:19Z

@phall1 Can you check with someone in DLC if transformers 4.29 is supported with PT 2.0. If not, can we close this PR?

oleksiishevchuk · 2023-08-23T18:27:29Z

It would be great to get some feedback about this, as we would like to add support for further transformers version (e.g. #4076)

mufaddal-rohawala

/bot run all

sagemaker-bot · 2023-10-26T01:18:39Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-local-mode-tests
Commit ID: a494586
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-10-26T01:33:54Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-pr
Commit ID: a494586
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-10-26T01:49:39Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-notebook-tests
Commit ID: a494586
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-10-26T01:56:52Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-slow-tests
Commit ID: a494586
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

sagemaker-bot · 2023-10-26T02:10:35Z

AWS CodeBuild CI Report

CodeBuild project: sagemaker-python-sdk-unit-tests
Commit ID: a494586
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

oleksiishevchuk · 2023-11-01T11:04:21Z

Hi @knikure, seems like the same issue with not available image still exists. Do you have any suggestions how to get feedback from the DLC team (@phall1 @ShiboXing) or if we can somehow contribute to help to resolve it?

knikure · 2023-11-02T15:21:31Z

Hi @knikure, seems like the same issue with not available image still exists. Do you have any suggestions how to get feedback from the DLC team (@phall1 @ShiboXing) or if we can somehow contribute to help to resolve it?

@oleksiishevchuk It would be better to always take a look at what DLC has already released for training and inference and raise issues like #3896 with DLC to make them cater to these requests. PySDK is downstream and cannot help release these images.

knikure · 2024-03-08T06:16:07Z

Closing this PR as we are tracking this as a feature request.

phall1 requested a review from a team as a code owner June 6, 2023 02:01

phall1 requested review from knikure and removed request for a team June 6, 2023 02:01

phall1 changed the title ~~Add transformers 4.29 - PyTorch 2.0.0 training and inference~~ feat: Add transformers 4.29 - PyTorch 2.0.0 training and inference Jun 6, 2023

knikure linked an issue Jun 6, 2023 that may be closed by this pull request

Add transformers v4.29.0 compatibility to sagemaker.huggingface #3896

Closed

knikure self-assigned this Jun 6, 2023

knikure force-pushed the update-hf4.29-pt2.0 branch from c061445 to 9be32b3 Compare June 6, 2023 06:01

knikure reviewed Jun 6, 2023

View reviewed changes

phall1 added a commit to phall1/sagemaker-python-sdk that referenced this pull request Jun 6, 2023

Update huggingface_training_pytorch_latest_version to py310

45c7e00

per the build errors in these tests: aws#3904 (comment) aws#3904 (comment)

knikure pushed a commit to phall1/sagemaker-python-sdk that referenced this pull request Jun 7, 2023

Update huggingface_training_pytorch_latest_version to py310

a84574b

per the build errors in these tests: aws#3904 (comment) aws#3904 (comment)

knikure force-pushed the update-hf4.29-pt2.0 branch from 45c7e00 to a84574b Compare June 7, 2023 04:23

mufaddal-rohawala reviewed Jun 7, 2023

View reviewed changes

knikure force-pushed the update-hf4.29-pt2.0 branch from 018e17d to 2b1464f Compare June 16, 2023 17:10

knikure reviewed Jun 16, 2023

View reviewed changes

oleksiishevchuk mentioned this pull request Aug 29, 2023

feature: Add transformers 4.32.0 - PyTorch 2.0.0 #4076

Closed

9 tasks

goelakash force-pushed the update-hf4.29-pt2.0 branch from 2b1464f to 1ec23be Compare October 9, 2023 23:10

phall1 added 3 commits October 16, 2023 09:22

Rebase on 4.28 change

8b0ec82

fix conftest conflicts

9523e72

Fix conftest if elif block

a494586

knikure force-pushed the update-hf4.29-pt2.0 branch from 1ec23be to a494586 Compare October 16, 2023 16:22

mufaddal-rohawala reviewed Oct 26, 2023

View reviewed changes

mufaddal-rohawala assigned benieric and unassigned knikure Oct 26, 2023

knikure closed this Mar 8, 2024

feat: Add transformers 4.29 - PyTorch 2.0.0 training and inference #3904

feat: Add transformers 4.29 - PyTorch 2.0.0 training and inference #3904

Conversation

phall1 commented Jun 6, 2023 • edited Loading

Merge Checklist

General

Tests

knikure left a comment • edited Loading

Choose a reason for hiding this comment

sagemaker-bot commented Jun 6, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Jun 6, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Jun 6, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Jun 6, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Jun 6, 2023

AWS CodeBuild CI Report

codecov-commenter commented Jun 6, 2023 • edited Loading

Codecov Report

phall1 commented Jun 6, 2023

sagemaker-bot commented Jun 7, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Jun 7, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Jun 7, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Jun 7, 2023

AWS CodeBuild CI Report

phall1 commented Jun 7, 2023

sagemaker-bot commented Jun 7, 2023

AWS CodeBuild CI Report

mufaddal-rohawala left a comment

Choose a reason for hiding this comment

sagemaker-bot commented Jun 7, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Jun 7, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Jun 7, 2023

AWS CodeBuild CI Report

knikure left a comment

Choose a reason for hiding this comment

sagemaker-bot commented Jun 16, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Jun 16, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Jun 16, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Jun 16, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Jun 16, 2023

AWS CodeBuild CI Report

phall1 commented Jun 16, 2023

knikure commented Jun 16, 2023

knikure commented Aug 14, 2023

oleksiishevchuk commented Aug 23, 2023

mufaddal-rohawala left a comment

Choose a reason for hiding this comment

sagemaker-bot commented Oct 26, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Oct 26, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Oct 26, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Oct 26, 2023

AWS CodeBuild CI Report

sagemaker-bot commented Oct 26, 2023

AWS CodeBuild CI Report

oleksiishevchuk commented Nov 1, 2023

knikure commented Nov 2, 2023

knikure commented Mar 8, 2024

phall1 commented Jun 6, 2023 •

edited

Loading

knikure left a comment •

edited

Loading

codecov-commenter commented Jun 6, 2023 •

edited

Loading