Fix ci image versions #129

suzhoum · 2024-06-07T06:05:45Z

Issue #, if available:

Description of changes:
This PR fixes a few bugs in CI.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

tonyhoo · 2024-06-07T18:09:01Z

.github/workflows/continuous_integration.yml

@@ -152,7 +152,7 @@ jobs:
          filters: |
            other_than_docs:
              - '!(docs/**)**'
-      - name: Test Tabular Cloud
+      - name: Test Text Cloud


tonyhoo · 2024-06-07T18:09:50Z

src/autogluon/cloud/backend/ray_backend.py

@@ -395,6 +395,10 @@ def _get_image_uri(self, framework_version: str, instance_type: str, custom_imag
                image_scope="training",
                instance_type=instance_type,
            )
+            if instance_type == "gpu":


Can you explain why this is needed?

This is just for testing, I created a few images with the correct ray version. I can remove them before merging.

tonyhoo · 2024-06-07T18:16:07Z

src/autogluon/cloud/default_cluster_configs/ray_aws_default_cluster_config.yaml

@@ -44,3 +44,4 @@ head_node_type: head
 initialization_commands:
  # To get access to DLC image
  - aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 763104351884.dkr.ecr.us-east-1.amazonaws.com
+  - aws ecr get-login-password --region us-west-2 | docker login --username AWS --password-stdin 763104351884.dkr.ecr.us-west-2.amazonaws.com


Do we need this region support with us-east-1 image

If user uses images before 0.8.0, I think they are hosted in us-east-1

If that's the case, shall we host all images in the same region say "us-east-1"?

tonyhoo · 2024-06-07T18:16:39Z

tests/conftest.py

-    gpu_training_image = "369469875935.dkr.ecr.us-east-1.amazonaws.com/autogluon-nightly-training:gpu-latest"
-    cpu_inference_image = "369469875935.dkr.ecr.us-east-1.amazonaws.com/autogluon-nightly-inference:cpu-latest"
-    gpu_inference_image = "369469875935.dkr.ecr.us-east-1.amazonaws.com/autogluon-nightly-inference:gpu-latest"
+    cpu_training_image = "369469875935.dkr.ecr.us-east-1.amazonaws.com/weisu:cpu-latest"


Why we need to use weisu's image instead of fixing the nightly image?

Sorry this is for testing. I created those just now.

tonyhoo · 2024-06-07T18:16:48Z

tests/unittests/cluster/test_distributed_training.py

            backend_kwargs={
                "initialization_commands": [
                    "aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 763104351884.dkr.ecr.us-east-1.amazonaws.com",
+                    "aws ecr get-login-password --region us-west-2 | docker login --username AWS --password-stdin 763104351884.dkr.ecr.us-west-2.amazonaws.com",


github-actions · 2024-06-08T06:27:21Z

Job PR-129-7d9fda5 is done.
Docs are uploaded to https://d12sc05jpx1wj5.cloudfront.net/PR-129/7d9fda5/index.html

github-actions · 2024-06-08T07:15:29Z

Job PR-129-4ea70d5 is done.
Docs are uploaded to https://d12sc05jpx1wj5.cloudfront.net/PR-129/4ea70d5/index.html

github-actions · 2024-06-09T07:41:15Z

Job PR-129-737a840 is done.
Docs are uploaded to https://d12sc05jpx1wj5.cloudfront.net/PR-129/737a840/index.html

suzhoum · 2024-06-10T14:44:21Z

@tonyhoo @prateekdesai04 CI is green after limiting the ray install in DLC images to align with what we have in https://github.com/autogluon/autogluon/blob/4825cff578c1cfad8997bd8b62dcd5749d0e55f1/core/setup.py#L51.
One issue I'm not sure about is that test_cluster always fails when first run on push https://github.com/autogluon/autogluon-cloud/actions/runs/9443270050 due to permission issue. However, when rerun the failed test in Action, the test passes https://github.com/autogluon/autogluon-cloud/actions/runs/9434703352.

tonyhoo · 2024-06-10T17:23:11Z

src/autogluon/cloud/backend/ray_backend.py

@@ -395,6 +395,10 @@ def _get_image_uri(self, framework_version: str, instance_type: str, custom_imag
                image_scope="training",
                instance_type=instance_type,
            )
+            if "g4dn" in instance_type:


The GPU image shall be applicable to all GPU instances. We shall update the logics here to be more robust for all GPU instance types:

def is_gpu_instance(instance_type): # A list of instance types that have GPUs gpu_instances = [ 'g2', 'g3', 'g4', 'p2', 'p3', 'p4', 'p4d', 'p5', 'p3dn', 'g4dn', 'g5', 'g5g' ] return any(instance_type.startswith(prefix) for prefix in gpu_instances) def select_image(): instance_type = get_instance_type() if is_gpu_instance(instance_type): return "369469875935.dkr.ecr.us-east-1.amazonaws.com/weisu:gpu-latest" else: return "369469875935.dkr.ecr.us-east-1.amazonaws.com/weisu:cpu-latest" ```

I will actually remove the whole commit 6cc23a7. This is to override the logic of getting the versioned DLC images using the sagemaker api for testing my custom images

tonyhoo · 2024-06-10T17:23:58Z

src/autogluon/cloud/default_cluster_configs/ray_aws_default_cluster_config.yaml

@@ -6,7 +6,7 @@ max_workers: 1

 docker:
    # The image uri will be dynamically replaced by cloud predictor
-    image: "763104351884.dkr.ecr.us-east-1.amazonaws.com/autogluon-training:0.7.0-cpu-py39-ubuntu20.04"
+    image: "763104351884.dkr.ecr.us-west-2.amazonaws.com/autogluon-training:1.1.0-cpu-py310-ubuntu20.04"


I would recommend to host the image to us-east-1 if not there already

I believe the image is already present in east-1

tonyhoo · 2024-06-10T17:24:22Z

src/autogluon/cloud/utils/ag_sagemaker.py

@@ -39,6 +39,10 @@ def __init__(
                image_scope="training",
                instance_type=instance_type,
            )
+            if "g4dn" in instance_type:


Same here, let's consolidate the image selection logic in a common function

same here, I will drop the commit.

suzhoum added 4 commits June 7, 2024 06:01

cluster test should pull versioned DLC

e069595

accommodate possible DLC region change

75f15a0

unpin framework version for multimodal_image test

b882165

fix framework version not passed in test

5892e42

suzhoum force-pushed the fix_ci_image_versions branch from eb4754b to 5892e42 Compare June 7, 2024 06:07

suzhoum requested a review from tonyhoo June 7, 2024 06:08

remove mmcv install

55c3169

tonyhoo reviewed Jun 7, 2024

View reviewed changes

suzhoum force-pushed the fix_ci_image_versions branch from ae4a28d to 7d9fda5 Compare June 8, 2024 05:45

suzhoum force-pushed the fix_ci_image_versions branch from 7d9fda5 to 4ea70d5 Compare June 8, 2024 06:36

suzhoum force-pushed the fix_ci_image_versions branch from 9247508 to 737a840 Compare June 9, 2024 06:52

suzhoum force-pushed the fix_ci_image_versions branch from 737a840 to 1d1811f Compare June 10, 2024 06:04

suzhoum requested a review from prateekdesai04 June 10, 2024 14:39

tonyhoo reviewed Jun 10, 2024

View reviewed changes

suzhoum added 3 commits June 10, 2024 18:57

increase wait time

307f93c

add more buffer time

c9e2ece

update region

4b9c762

suzhoum force-pushed the fix_ci_image_versions branch from 1d1811f to 4b9c762 Compare June 10, 2024 19:00

add delay after aws credentials

db885de

suzhoum added the safe to test This PR is safe to be tested in the cloud label Jun 10, 2024

github-actions bot removed the safe to test This PR is safe to be tested in the cloud label Jun 10, 2024

fix matrix

77e405f

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix ci image versions #129

Fix ci image versions #129

suzhoum commented Jun 7, 2024

tonyhoo Jun 7, 2024

tonyhoo Jun 7, 2024

suzhoum Jun 7, 2024

tonyhoo Jun 7, 2024

suzhoum Jun 7, 2024

tonyhoo Jun 10, 2024

tonyhoo Jun 7, 2024

suzhoum Jun 7, 2024

tonyhoo Jun 7, 2024

github-actions bot commented Jun 8, 2024

github-actions bot commented Jun 8, 2024

github-actions bot commented Jun 9, 2024

suzhoum commented Jun 10, 2024 •

edited

Loading

tonyhoo Jun 10, 2024

suzhoum Jun 10, 2024 •

edited

Loading

tonyhoo Jun 10, 2024

prateekdesai04 Jun 10, 2024 •

edited

Loading

suzhoum Jun 10, 2024

tonyhoo Jun 10, 2024

suzhoum Jun 10, 2024

Fix ci image versions #129

Are you sure you want to change the base?

Fix ci image versions #129

Conversation

suzhoum commented Jun 7, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Jun 8, 2024

github-actions bot commented Jun 8, 2024

github-actions bot commented Jun 9, 2024

suzhoum commented Jun 10, 2024 • edited Loading

Choose a reason for hiding this comment

suzhoum Jun 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

prateekdesai04 Jun 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

suzhoum commented Jun 10, 2024 •

edited

Loading

suzhoum Jun 10, 2024 •

edited

Loading

prateekdesai04 Jun 10, 2024 •

edited

Loading