[Resources] Add `cpus` in resource specification #1622

WoosukKwon · 2023-01-24T08:49:52Z

Closes #387

This PR introduces the ability for users to specify resource requirements in terms of the number of vCPUs in the Resources class and the resources section in the SkyPilot yaml.

Semantics

If the instance type and accelerators are not specified, SkyPilot will use the cpus parameter to select an appropriate VM size from the cloud's default VM family (i.e., m6i for AWS, Dv5 for Azure, and n2-standard for GCP).

For example,

sky launch --cpus 8 or Resources(cpus=8): SkyPilot will find the VMs that have exactly 8 vCPUs. Namely,
- AWS: m6i.2xlarge
- Azure: Standard_D8_v5
- GCP: n2-standard-8
sky launch --cpus 12+ or Resources(cpus='12+'): SkyPilot will consider the VMs that have 12 or more vCPUs. Namely,
- AWS: m6i.4xlarge
- Azure: Standard_D16_v5
- GCP: n2-standard-16

(Note: The selected three instance types have 16 vCPUs instead of 12 vCPUs, because the default instance families of the clouds do not include any VM with 12-15 vCPUs.)

If the accelerators are specified, the instance family can be inferred for each cloud, and SkyPilot will use the cpus parameter to choose the VM of the right size in the instance family. For example,

sky launch --gpus T4 --cpus 8+ or Resources(accelerators='T4', cpus='8+'): SkyPilot will choose VMs that have 1 T4 GPU along with 8 or more vCPUs. Namely,
- AWS: g4dn.2xlarge'
- Azure: Standard_NC8as_T4_v3
- GCP: n1-highmem-8 (+ T4)

The same semantics applies to cpunode, gpunode, tpunode, and spot launch.

Tested (run the relevant ones):

Any manual or new tests for this PR (please specify below)
All smoke tests: pytest tests/test_smoke.py
Relevant individual smoke tests: pytest tests/test_smoke.py::test_fill_in_the_name
Backward compatibility tests: bash tests/backward_comaptibility_tests.sh

concretevitamin

This is awesome @WoosukKwon! Had a quick scan and the approach makes sense to me.

Quick question: are we deferring adding CPU as a scheduling constraint?

E.g., what happens if we launch a node with 8 CPUs, then keep sky launch -c existing --cpu 4 for many times? Or, for --cpu 16 do we correctly throw an error?

(I think it's okay to defer, just checking.)

sky/clouds/service_catalog/aws_catalog.py

WoosukKwon · 2023-01-24T17:37:57Z

@concretevitamin Thanks for the quick feedback. Yes, I'd like to defer it.

dongreenberg · 2023-01-24T18:21:39Z

Love it!! Two questions:

I'm assuming this will work with mixing and matching inside the Resources() python API? e.g. I could do
Resources(accelerators='A100:1', cpus='8+')?
Do all of the CPU-only boxes launch with the same OS and architecture?

WoosukKwon · 2023-01-24T19:03:32Z

Hey @dongreenberg, thanks for your questions!

I'm assuming this will work with mixing and matching inside the Resources() python API? e.g. I could do
Resources(accelerators='A100:1', cpus='8+')?

Yes. It will work on the Python API in the same way.

Do all of the CPU-only boxes launch with the same OS and architecture?

They all have recent Intel x86-64 CPUs (i.e., either Ice Lake or Cascade Lake CPUs), and they all have 4 GB RAM per 1 vCPU. More specifically,

AWS m6i series have Intel Ice Lake 8375C CPUs
Azure Standard Dv5 series have Ice Lake 8370C CPUs
GCP N2 series have Intel Ice Lake 8373C or Cascade Lake 6268CL CPUs

The OS might be different across the clouds and the VM images you select.

dongreenberg · 2023-01-24T19:04:40Z

Amazing, that's perfect. You guys are the best!

concretevitamin

Very nice @WoosukKwon. Did a pass and left some comments.

Tests: consider spelling out the manual tests. Also, should we add some dryrun unit tests to make sure sky.optimize() works for things like sky.Resources(..., cpu=...)?
Should we warn + ignore --cpu in sky launch -c existing --cpu x?

sky/cli.py

sky/optimizer.py

sky/resources.py

sky/clouds/service_catalog/azure_catalog.py

sky/execution.py

sky/clouds/service_catalog/gcp_catalog.py

concretevitamin

Thanks @WoosukKwon. Some final comments & reminder to add some tests to maybe test_optimizer_dryruns.py.

concretevitamin · 2023-01-26T16:44:34Z

sky/resources.py

-    # 2. Change the __setstate__ method to handle the new fields.
-    # 3. Modify the to_config method to handle the new fields.
+    # If any fields changed, increment the version. For backward compatibility,
+    # modify the __setstate__ method to handle the old version.


Makes sense. Was thinking this comment is a guide for "all the places we need to change when we add a new field". For just back compat, it seems __setstate__ is the only required change. cc @Michaelvll to double check

concretevitamin · 2023-01-26T16:45:50Z

sky/clouds/aws.py

-            return (_make([AWS.get_default_instance_type()]),
-                    fuzzy_candidate_list)
+            # Return a default instance type with the given number of vCPUs.
+            default_instance_type = AWS.get_default_instance_type(


Do you mind clarifying what you meant?

Currently, the Cloud interface includes a shallow method, which does not show the default type:

@classmethod def get_default_instance_type(cls, cpu: Optional[str] = None) -> Optional[str]: return service_catalog.get_default_instance_type(cpu=cpu, clouds='aws')

concretevitamin

Thanks for the awesome contribution @WoosukKwon. A major update to our Resources in a long time. LGTM.

PS: consider some basic smoke tests (e.g., minimal) before merging?

tests/test_optimizer_dryruns.py

WoosukKwon · 2023-01-28T07:22:43Z

@concretevitamin This PR has passed all the smoke tests.

concretevitamin · 2023-01-28T15:41:56Z

Missed this before: shall we add the new field to https://skypilot.readthedocs.io/en/latest/reference/yaml-spec.html?

WoosukKwon · 2023-01-28T20:18:21Z

Good catch. Added.

dongreenberg · 2023-01-30T15:42:56Z

Woo!!! Congrats and thanks gang!! Can't wait to test it out!

WoosukKwon added 11 commits January 23, 2023 23:30

Add cpu option

d5b5511

yapf

3e0fa4a

Add cpu in CLI

22fe74b

Fix GCP and Do not raise errors

386b1dc

yapf

23a0ea5

Fix TPU VM

28b0c5a

Print error msg for cpu mismatches

0d8cf90

Fix error when default instance is not found

a825cf7

Merge branch 'master' into woosuk-cpu

d65634c

Fix a comment

4959f8f

Minor

5d57fd5

WoosukKwon requested a review from concretevitamin January 24, 2023 09:32

WoosukKwon changed the title ~~[Resources] Add cpu in resources~~ [Resources] Add cpu in resource specification Jan 24, 2023

concretevitamin reviewed Jan 24, 2023

View reviewed changes

sky/clouds/service_catalog/aws_catalog.py Outdated Show resolved Hide resolved

WoosukKwon added 3 commits January 24, 2023 09:41

Add _impl

9067308

Define _DEFAULT_NUM_VCPUS as constant

797a632

yapf

b4e0b00

WoosukKwon added 2 commits January 24, 2023 13:26

Ban specifying CPU in sky exec

b95c56b

yapf

9324e52

concretevitamin reviewed Jan 25, 2023

View reviewed changes

WoosukKwon added 6 commits January 24, 2023 20:02

Fix type error

12f4bbe

xlarge -> large

09ad032

Fix AWS

83a7a9d

Fix comments on default instance family

c400dea

Fix sky exec

75cf94a

minor

afd80fe

WoosukKwon added 3 commits January 25, 2023 15:31

Fix docstring

8f9dabc

Add assert in less_demanding_than

97dc60e

roll back

4599f36

WoosukKwon requested a review from concretevitamin January 25, 2023 23:41

concretevitamin reviewed Jan 26, 2023

View reviewed changes

WoosukKwon added 9 commits January 27, 2023 10:53

Merge branch 'master' into woosuk-cpu

01021c9

Remove is_same_resources

4949000

Add TODO

fec1b7a

Check cpus format again in _filter_with_cpus

13607fd

yapf

2059309

Add tests

6a2b554

Fix tests

99a66e7

Fix

ad4827b

Fix

2c08184

WoosukKwon requested a review from concretevitamin January 27, 2023 20:49

concretevitamin approved these changes Jan 27, 2023

View reviewed changes

tests/test_optimizer_dryruns.py Outdated Show resolved Hide resolved

WoosukKwon added 3 commits January 27, 2023 22:16

Fix tests

014f63c

Merge branch 'master' into woosuk-cpu

1f8a031

Fix example_app

62aa150

concretevitamin approved these changes Jan 28, 2023

View reviewed changes

Add cpus in yaml spec

4ea39d7

WoosukKwon merged commit 76eed62 into master Jan 28, 2023

WoosukKwon deleted the woosuk-cpu branch January 28, 2023 23:54

sumanthgenz pushed a commit to sumanthgenz/skypilot that referenced this pull request Feb 22, 2023

[Resources] Add cpus in resource specification (skypilot-org#1622)

adb667b

Michaelvll mentioned this pull request Mar 8, 2023

[Resources] Add memory in resources #1746

Merged

5 tasks

sumanthgenz pushed a commit to sumanthgenz/skypilot that referenced this pull request Mar 15, 2023

[Resources] Add cpus in resource specification (skypilot-org#1622)

a50a357

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Resources] Add `cpus` in resource specification #1622

[Resources] Add `cpus` in resource specification #1622

WoosukKwon commented Jan 24, 2023 •

edited

concretevitamin left a comment

WoosukKwon commented Jan 24, 2023

dongreenberg commented Jan 24, 2023 •

edited

WoosukKwon commented Jan 24, 2023

dongreenberg commented Jan 24, 2023

concretevitamin left a comment

concretevitamin left a comment

concretevitamin Jan 26, 2023

concretevitamin Jan 26, 2023

concretevitamin left a comment

WoosukKwon commented Jan 28, 2023

concretevitamin commented Jan 28, 2023

WoosukKwon commented Jan 28, 2023

dongreenberg commented Jan 30, 2023

[Resources] Add cpus in resource specification #1622

[Resources] Add cpus in resource specification #1622

Conversation

WoosukKwon commented Jan 24, 2023 • edited

concretevitamin left a comment

Choose a reason for hiding this comment

WoosukKwon commented Jan 24, 2023

dongreenberg commented Jan 24, 2023 • edited

WoosukKwon commented Jan 24, 2023

dongreenberg commented Jan 24, 2023

concretevitamin left a comment

Choose a reason for hiding this comment

concretevitamin left a comment

Choose a reason for hiding this comment

concretevitamin Jan 26, 2023

Choose a reason for hiding this comment

concretevitamin Jan 26, 2023

Choose a reason for hiding this comment

concretevitamin left a comment

Choose a reason for hiding this comment

WoosukKwon commented Jan 28, 2023

concretevitamin commented Jan 28, 2023

WoosukKwon commented Jan 28, 2023

dongreenberg commented Jan 30, 2023

[Resources] Add `cpus` in resource specification #1622

[Resources] Add `cpus` in resource specification #1622

WoosukKwon commented Jan 24, 2023 •

edited

dongreenberg commented Jan 24, 2023 •

edited