Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add commit and push options to dev env script #4015

Merged
merged 47 commits into from
Jun 25, 2024
Merged

Conversation

arjkesh
Copy link
Contributor

@arjkesh arjkesh commented Jun 19, 2024

GitHub Issue #, if available:

Note:

  • If merging this PR should also close the associated Issue, please also add that Issue # to the Linked Issues section on the right.

  • All PR's are checked weekly for staleness. This PR will be closed if not updated in 30 days.

Description

Add commit and push feature to the dev env preparer

Tests run

Run existing unit tests, test in commit thread below

NOTE: By default, docker builds are disabled. In order to build your container, please update dlc_developer_config.toml and specify the framework to build in "build_frameworks"

  • I have run builds/tests on commit for my changes.

NOTE: If you are creating a PR for a new framework version, please ensure success of the standard, rc, and efa sagemaker remote tests by updating the dlc_developer_config.toml file:

Expand
  • sagemaker_remote_tests = true
  • sagemaker_efa_tests = true
  • sagemaker_rc_tests = true

Additionally, please run the sagemaker local tests in at least one revision:

  • sagemaker_local_tests = true

Formatting

DLC image/dockerfile

Builds to Execute

Expand

Fill out the template and click the checkbox of the builds you'd like to execute

Note: Replace with <X.Y> with the major.minor framework version (i.e. 2.2) you would like to start.

  • build_pytorch_training_<X.Y>_sm

  • build_pytorch_training_<X.Y>_ec2

  • build_pytorch_inference_<X.Y>_sm

  • build_pytorch_inference_<X.Y>_ec2

  • build_pytorch_inference_<X.Y>_graviton

  • build_tensorflow_training_<X.Y>_sm

  • build_tensorflow_training_<X.Y>_ec2

  • build_tensorflow_inference_<X.Y>_sm

  • build_tensorflow_inference_<X.Y>_ec2

  • build_tensorflow_inference_<X.Y>_graviton

Additional context

PR Checklist

Expand
  • I've prepended PR tag with frameworks/job this applies to : [mxnet, tensorflow, pytorch] | [ei/neuron/graviton] | [build] | [test] | [benchmark] | [ec2, ecs, eks, sagemaker]
  • If the PR changes affects SM test, I've modified dlc_developer_config.toml in my PR branch by setting sagemaker_tests = true and efa_tests = true
  • If this PR changes existing code, the change fully backward compatible with pre-existing code. (Non backward-compatible changes need special approval.)
  • (If applicable) I've documented below the DLC image/dockerfile this relates to
  • (If applicable) I've documented below the tests I've run on the DLC image
  • (If applicable) I've reviewed the licenses of updated and new binaries and their dependencies to make sure all licenses are on the Apache Software Foundation Third Party License Policy Category A or Category B license list. See https://www.apache.org/legal/resolved.html.
  • (If applicable) I've scanned the updated and new binaries to make sure they do not have vulnerabilities associated with them.

NEURON/GRAVITON Testing Checklist

  • When creating a PR:
  • I've modified dlc_developer_config.toml in my PR branch by setting neuron_mode = true or graviton_mode = true

Benchmark Testing Checklist

  • When creating a PR:
  • I've modified dlc_developer_config.toml in my PR branch by setting ec2_benchmark_tests = true or sagemaker_benchmark_tests = true

Pytest Marker Checklist

Expand
  • (If applicable) I have added the marker @pytest.mark.model("<model-type>") to the new tests which I have added, to specify the Deep Learning model that is used in the test (use "N/A" if the test doesn't use a model)
  • (If applicable) I have added the marker @pytest.mark.integration("<feature-being-tested>") to the new tests which I have added, to specify the feature that will be tested
  • (If applicable) I have added the marker @pytest.mark.multinode(<integer-num-nodes>) to the new tests which I have added, to specify the number of nodes used on a multi-node test
  • (If applicable) I have added the marker @pytest.mark.processor(<"cpu"/"gpu"/"eia"/"neuron">) to the new tests which I have added, if a test is specifically applicable to only one processor type

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

dlc_developer_config.toml
{   'build': {   'build_frameworks': ['pytorch'],
                 'build_inference': True,
                 'build_training': True},
    'buildspec_override': {   'dlc-pr-pytorch-inference': 'pytorch/inference/buildspec-2-2-ec2.yml',
                              'dlc-pr-pytorch-training': 'pytorch/training/buildspec-2-2-sm.yml'},
    'dev': {   'deep_canary_mode': False,
               'graviton_mode': False,
               'neuronx_mode': False},
    'test': {   'ec2_tests': True,
                'ecs_tests': False,
                'eks_tests': False,
                'sagemaker_local_tests': False,
                'sagemaker_remote_tests': False,
                'sanity_tests': True}}
@aws-deep-learning-containers-ci aws-deep-learning-containers-ci bot added Size:S Determines the size of the PR src Reflects file change in src folder labels Jun 19, 2024
@arjkesh arjkesh changed the title Tester3 Testing adding commit feature Jun 19, 2024
dlc_developer_config.toml
{   'build': {   'build_frameworks': ['pytorch'],
                 'build_inference': True,
                 'build_training': True},
    'buildspec_override': {   'dlc-pr-pytorch-inference': 'pytorch/inference/buildspec-2-2-ec2.yml',
                              'dlc-pr-pytorch-training': 'pytorch/training/buildspec-2-2-sm.yml'},
    'dev': {   'deep_canary_mode': False,
               'graviton_mode': False,
               'neuronx_mode': False},
    'test': {   'ec2_tests': True,
                'ecs_tests': False,
                'eks_tests': False,
                'sagemaker_local_tests': False,
                'sagemaker_remote_tests': False,
                'sanity_tests': True}}
dlc_developer_config.toml
{   'build': {   'build_frameworks': ['pytorch'],
                 'build_inference': True,
                 'build_training': True},
    'buildspec_override': {   'dlc-pr-pytorch-inference': 'pytorch/inference/buildspec-2-2-ec2.yml',
                              'dlc-pr-pytorch-training': 'pytorch/training/buildspec-2-2-sm.yml'},
    'dev': {   'deep_canary_mode': False,
               'graviton_mode': False,
               'neuronx_mode': False},
    'test': {   'ec2_tests': True,
                'ecs_tests': False,
                'eks_tests': False,
                'sagemaker_local_tests': False,
                'sagemaker_remote_tests': False,
                'sanity_tests': True}}
dlc_developer_config.toml
{   'build': {   'build_frameworks': ['pytorch'],
                 'build_inference': True,
                 'build_training': True},
    'buildspec_override': {   'dlc-pr-pytorch-inference': 'pytorch/inference/buildspec-2-2-ec2.yml',
                              'dlc-pr-pytorch-training': 'pytorch/training/buildspec-2-2-sm.yml'},
    'dev': {   'deep_canary_mode': False,
               'graviton_mode': False,
               'neuronx_mode': False},
    'test': {   'ec2_tests': True,
                'ecs_tests': False,
                'eks_tests': False,
                'sagemaker_local_tests': False,
                'sagemaker_remote_tests': False,
                'sanity_tests': True}}
dlc_developer_config.toml
{   'build': {   'build_frameworks': ['pytorch'],
                 'build_inference': True,
                 'build_training': True},
    'buildspec_override': {   'dlc-pr-pytorch-inference': 'pytorch/inference/buildspec-2-2-ec2.yml',
                              'dlc-pr-pytorch-training': 'pytorch/training/buildspec-2-2-sm.yml'},
    'dev': {   'deep_canary_mode': False,
               'graviton_mode': False,
               'neuronx_mode': False},
    'test': {   'ec2_tests': True,
                'ecs_tests': False,
                'eks_tests': False,
                'sagemaker_local_tests': False,
                'sagemaker_remote_tests': False,
                'sanity_tests': True}}
dlc_developer_config.toml
{   'build': {   'build_frameworks': ['pytorch'],
                 'build_inference': True,
                 'build_training': True},
    'buildspec_override': {   'dlc-pr-pytorch-inference': 'pytorch/inference/buildspec-2-2-ec2.yml',
                              'dlc-pr-pytorch-training': 'pytorch/training/buildspec-2-2-sm.yml'},
    'dev': {   'deep_canary_mode': False,
               'graviton_mode': False,
               'neuronx_mode': False},
    'test': {   'ec2_tests': True,
                'ecs_tests': False,
                'eks_tests': False,
                'sagemaker_local_tests': False,
                'sagemaker_remote_tests': False,
                'sanity_tests': True}}
dlc_developer_config.toml
{   'build': {   'build_frameworks': ['pytorch'],
                 'build_inference': True,
                 'build_training': True},
    'buildspec_override': {   'dlc-pr-pytorch-inference': 'pytorch/inference/buildspec-2-2-ec2.yml',
                              'dlc-pr-pytorch-training': 'pytorch/training/buildspec-2-2-sm.yml'},
    'dev': {   'deep_canary_mode': False,
               'graviton_mode': False,
               'neuronx_mode': False},
    'test': {   'ec2_tests': True,
                'ecs_tests': False,
                'eks_tests': False,
                'sagemaker_local_tests': False,
                'sagemaker_remote_tests': False,
                'sanity_tests': True}}
dlc_developer_config.toml
{'build': {'build_frameworks': ['pytorch'],
           'build_inference': True,
           'build_training': True},
 'buildspec_override': {'dlc-pr-pytorch-inference': 'pytorch/inference/buildspec-2-2-ec2.yml',
                        'dlc-pr-pytorch-training': 'pytorch/training/buildspec-2-2-sm.yml'},
 'dev': {'deep_canary_mode': False,
         'graviton_mode': False,
         'neuronx_mode': False},
 'test': {'ec2_tests': True,
          'ecs_tests': False,
          'eks_tests': False,
          'sagemaker_local_tests': False,
          'sagemaker_remote_tests': False,
          'sanity_tests': True}}
dlc_developer_config.toml
{'build': {'build_frameworks': ['pytorch'],
           'build_inference': True,
           'build_training': True},
 'buildspec_override': {'dlc-pr-pytorch-inference': 'pytorch/inference/buildspec-2-2-ec2.yml',
                        'dlc-pr-pytorch-training': 'pytorch/training/buildspec-2-2-sm.yml'},
 'dev': {'deep_canary_mode': False,
         'graviton_mode': False,
         'neuronx_mode': False},
 'test': {'ec2_tests': True,
          'ecs_tests': False,
          'eks_tests': False,
          'sagemaker_local_tests': False,
          'sagemaker_remote_tests': False,
          'sanity_tests': True}}
dlc_developer_config.toml
{'build': {'build_frameworks': ['pytorch'],
           'build_inference': True,
           'build_training': True},
 'buildspec_override': {'dlc-pr-pytorch-inference': 'pytorch/inference/buildspec-2-2-ec2.yml',
                        'dlc-pr-pytorch-training': 'pytorch/training/buildspec-2-2-sm.yml'},
 'dev': {'deep_canary_mode': False,
         'graviton_mode': False,
         'neuronx_mode': False},
 'test': {'ec2_tests': True,
          'ecs_tests': False,
          'eks_tests': False,
          'sagemaker_local_tests': False,
          'sagemaker_remote_tests': False,
          'sanity_tests': True}}
dlc_developer_config.toml
{'build': {'build_frameworks': ['pytorch'],
           'build_inference': True,
           'build_training': True},
 'buildspec_override': {'dlc-pr-pytorch-inference': 'pytorch/inference/buildspec-2-2-ec2.yml',
                        'dlc-pr-pytorch-training': 'pytorch/training/buildspec-2-2-sm.yml'},
 'dev': {'deep_canary_mode': False,
         'graviton_mode': False,
         'neuronx_mode': False},
 'test': {'ec2_tests': True,
          'ecs_tests': False,
          'eks_tests': False,
          'sagemaker_local_tests': False,
          'sagemaker_remote_tests': False,
          'sanity_tests': True}}
dlc_developer_config.toml
{'build': {'build_frameworks': ['pytorch'],
           'build_inference': True,
           'build_training': True},
 'buildspec_override': {'dlc-pr-pytorch-inference': 'pytorch/inference/buildspec-2-2-ec2.yml',
                        'dlc-pr-pytorch-training': 'pytorch/training/buildspec-2-2-sm.yml'},
 'dev': {'deep_canary_mode': False,
         'graviton_mode': False,
         'neuronx_mode': False},
 'test': {'ec2_tests': True,
          'ecs_tests': False,
          'eks_tests': False,
          'sagemaker_local_tests': False,
          'sagemaker_remote_tests': False,
          'sanity_tests': True}}
dlc_developer_config.toml
{'build': {'build_frameworks': ['pytorch'],
           'build_inference': True,
           'build_training': True},
 'buildspec_override': {'dlc-pr-pytorch-inference': 'pytorch/inference/buildspec-2-2-ec2.yml',
                        'dlc-pr-pytorch-training': 'pytorch/training/buildspec-2-2-sm.yml'},
 'dev': {'deep_canary_mode': False,
         'graviton_mode': False,
         'neuronx_mode': False},
 'test': {'ec2_tests': True,
          'ecs_tests': False,
          'eks_tests': False,
          'sagemaker_local_tests': False,
          'sagemaker_remote_tests': False,
          'sanity_tests': True}}
dlc_developer_config.toml
{   'build': {   'build_frameworks': ['pytorch'],
                 'build_inference': True,
                 'build_training': True},
    'buildspec_override': {   'dlc-pr-pytorch-inference': 'pytorch/inference/buildspec-2-2-ec2.yml',
                              'dlc-pr-pytorch-training': 'pytorch/training/buildspec-2-2-sm.yml'},
    'dev': {   'deep_canary_mode': False,
               'graviton_mode': False,
               'neuronx_mode': False},
    'test': {   'ec2_tests': True,
                'ecs_tests': False,
                'eks_tests': False,
                'sagemaker_local_tests': False,
                'sagemaker_remote_tests': False,
                'sanity_tests': True}}
dlc_developer_config.toml
{   'build': {   'build_frameworks': ['pytorch'],
                 'build_inference': True,
                 'build_training': False},
    'buildspec_override': {   'dlc-pr-pytorch-neuronx-inference': 'pytorch/inference/buildspec-neuronx.yml'},
    'dev': {   'deep_canary_mode': False,
               'graviton_mode': False,
               'neuronx_mode': True},
    'test': {   'ec2_tests': True,
                'ecs_tests': False,
                'eks_tests': False,
                'sagemaker_local_tests': False,
                'sagemaker_remote_tests': False,
                'sanity_tests': True}}
dlc_developer_config.toml
{   'build': {   'build_frameworks': ['pytorch'],
                 'build_inference': True,
                 'build_training': False},
    'buildspec_override': {   'dlc-pr-pytorch-neuronx-inference': 'pytorch/inference/buildspec-neuronx.yml'},
    'dev': {   'deep_canary_mode': False,
               'graviton_mode': False,
               'neuronx_mode': True},
    'test': {   'ec2_tests': True,
                'ecs_tests': False,
                'eks_tests': False,
                'sagemaker_local_tests': False,
                'sagemaker_remote_tests': False,
                'sanity_tests': True}}
dlc_developer_config.toml
{   'build': {   'build_frameworks': ['pytorch'],
                 'build_inference': True,
                 'build_training': False},
    'buildspec_override': {   'dlc-pr-pytorch-neuronx-inference': 'pytorch/inference/buildspec-neuronx.yml'},
    'dev': {   'deep_canary_mode': False,
               'graviton_mode': False,
               'neuronx_mode': True},
    'test': {   'ec2_tests': True,
                'ecs_tests': False,
                'eks_tests': False,
                'sagemaker_local_tests': False,
                'sagemaker_remote_tests': False,
                'sanity_tests': True}}
dlc_developer_config.toml
{   'build': {   'build_frameworks': ['pytorch'],
                 'build_inference': True,
                 'build_training': False},
    'buildspec_override': {   'dlc-pr-pytorch-neuronx-inference': 'pytorch/inference/buildspec-neuronx.yml'},
    'dev': {   'deep_canary_mode': False,
               'graviton_mode': False,
               'neuronx_mode': True},
    'test': {   'ec2_tests': True,
                'ecs_tests': False,
                'eks_tests': False,
                'sagemaker_local_tests': False,
                'sagemaker_remote_tests': False,
                'sanity_tests': True}}
dlc_developer_config.toml
{   'build': {   'build_frameworks': ['pytorch'],
                 'build_inference': True,
                 'build_training': False},
    'buildspec_override': {   'dlc-pr-pytorch-neuronx-inference': 'pytorch/inference/buildspec-neuronx.yml'},
    'dev': {   'deep_canary_mode': False,
               'graviton_mode': False,
               'neuronx_mode': True},
    'test': {   'ec2_tests': True,
                'ecs_tests': False,
                'eks_tests': False,
                'sagemaker_local_tests': False,
                'sagemaker_remote_tests': False,
                'sanity_tests': True}}
dlc_developer_config.toml
{   'build': {   'build_frameworks': ['pytorch'],
                 'build_inference': True,
                 'build_training': False},
    'buildspec_override': {   'dlc-pr-pytorch-neuronx-inference': 'pytorch/inference/buildspec-neuronx.yml'},
    'dev': {   'deep_canary_mode': False,
               'graviton_mode': False,
               'neuronx_mode': True},
    'test': {   'ec2_tests': True,
                'ecs_tests': False,
                'eks_tests': False,
                'sagemaker_local_tests': False,
                'sagemaker_remote_tests': False,
                'sanity_tests': True}}
@aws-deep-learning-containers-ci aws-deep-learning-containers-ci bot added sanity Reflects file change in dlc_tests/sanity folder test Reflects file change in test folder labels Jun 20, 2024
@arjkesh arjkesh changed the title Testing adding commit feature Add commit and push options to dev env script Jun 20, 2024
@arjkesh arjkesh marked this pull request as ready for review June 20, 2024 17:56
@arjkesh arjkesh requested a review from a team as a code owner June 20, 2024 17:56
dlc_developer_config.toml
{   'build': {   'build_frameworks': ['pytorch'],
                 'build_inference': True,
                 'build_training': False},
    'buildspec_override': {   'dlc-pr-pytorch-neuronx-inference': 'pytorch/inference/buildspec-neuronx.yml'},
    'dev': {   'deep_canary_mode': False,
               'graviton_mode': False,
               'neuronx_mode': True},
    'test': {   'ec2_tests': True,
                'ecs_tests': False,
                'eks_tests': False,
                'sagemaker_local_tests': False,
                'sagemaker_remote_tests': False,
                'sanity_tests': True}}
@arjkesh arjkesh enabled auto-merge (squash) June 25, 2024 20:52
@arjkesh arjkesh merged commit 15eed9d into aws:master Jun 25, 2024
28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
sanity Reflects file change in dlc_tests/sanity folder Size:S Determines the size of the PR src Reflects file change in src folder test Reflects file change in test folder
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants