[Core] Support Multiple Resources

[MaoZiming]

This partially builds upon an old PR #1389.
The user can specify an ordered preference list or an unordered restriction set for accelerators.
Ordered preference list. The user prefers A100-40GB:1 over V100:1, etc. The optimizer will follow user-specified order in picking the accelerators.

accelerators: ['A100-40GB:1', 'V100:1', 'K80:1', 'T4:1']

Unordered restriction set. The user specifies multiple accelerators for accelerator to choose from, with no preference among them.

accelerators: {'A100-40GB:1', 'K80:1', 'V100:1', 'T4:1', 'T4:4'}

Tested (run the relevant ones):

• Code formatting: bash format.sh
• New manual tests: sky launch examples/multi_resources.yaml
• New manual tests: sky launch examples/multi_accelerators.yaml
• New smoke test: pytest tests/test_smoke.py::test_multiple_resources_unordered
• New smoke test: pytest tests/test_smoke.py::test_multiple_resources_ordered
• All smoke tests: pytest tests/test_smoke.py

[MaoZiming]

@infwinston We might need to think about the UI.
The optimizer will only print one chosen_resource per cloud. If the user specifies multiple accelerators, only one of them (picked by the optimizer) will be printed.

[concretevitamin]

This is awesome @MaoZiming. Tried it out real quick, some UX notes:

» sky launch --use-spot --down t-multi.yaml
Task from YAML spec: t-multi.yaml
Task(run=<empty>)
  resources: {<Cloud>([Spot], {'A100-80GB': 1}), <Cloud>([Spot], {'A100-40GB': 1})}
I 08-31 10:42:49 optimizer.py:129] Using user-specified resource list.
I 08-31 10:42:49 optimizer.py:1135] No resource satisfying <Cloud>([Spot], {'A100-40GB': 1}) on [AWS, Azure, GCP, Lambda, OCI, SCP].
I 08-31 10:42:49 optimizer.py:1135] No resource satisfying <Cloud>([Spot], {'A100-40GB': 1}) on [AWS, Azure, GCP, Lambda, OCI, SCP].
I 08-31 10:42:49 optimizer.py:703] == Optimizer ==
I 08-31 10:42:49 optimizer.py:715] Target: minimizing cost
I 08-31 10:42:49 optimizer.py:726] Estimated cost: $1.9 / hour
I 08-31 10:42:49 optimizer.py:726]
I 08-31 10:42:49 optimizer.py:800] Considered resources (1 node):
I 08-31 10:42:49 optimizer.py:855] ------------------------------------------------------------------------------------------------------
I 08-31 10:42:49 optimizer.py:855]  CLOUD   INSTANCE               vCPUs   Mem(GB)   ACCELERATORS   REGION/ZONE      COST ($)   CHOSEN
I 08-31 10:42:49 optimizer.py:855] ------------------------------------------------------------------------------------------------------
I 08-31 10:42:49 optimizer.py:855]  GCP     a2-ultragpu-1g[Spot]   12      170       A100-80GB:1    europe-west4-a   1.86          ✔
I 08-31 10:42:49 optimizer.py:855] ------------------------------------------------------------------------------------------------------
I 08-31 10:42:49 optimizer.py:855]
Launching a new cluster 'sky-957c-zongheng'. Proceed? [Y/n]:

I intentionally used an invalid GPU name and it worked out well by skipping it!

resources:
  accelerators: ['A100-40GB:1', 'A100-80GB']

1. This set-based repr is surprising, because previously we used a set to mean "pick the best of these candidates" (see example_app.py), rather than "try these sequentially":

Task(run=<empty>)
  resources: {<Cloud>([Spot], {'A100-80GB': 1}), <Cloud>([Spot], {'A100-40GB': 1})}

2. We may want to change this

I 08-31 10:42:49 optimizer.py:129] Using user-specified resource list. 

to bold/colored,

I 08-31 10:42:49 optimizer.py:129] Using user-specified accelerators list (will be tried in the listed order).

3. A bit surprising to see No resource satisfying <Cloud>([Spot], {'A100-40GB': 1}) on [AWS, Azure, GCP, Lambda, OCI, SCP]. being printed twice?

[MaoZiming]

@concretevitamin Thanks for the suggestions!
I have fixed 2 and 3. For 1, we are still deciding on the interface. We initially wanted to use
['A100-40GB:1', 'A100-80GB'] to mean a list of accelerators that will be considered sequentially.

[MaoZiming]

@infwinston hey this should be working with the spot controller. When you get some time maybe could you review the code? Thanks!

[Michaelvll]

Thanks for the update @MaoZiming! The code looks mostly good to me! Once we have done the test for the smoke test and some manual tests. I think it should be good to go in.

[Michaelvll]

I see, in that case, which of the schemas here represents the {V100:1, A100:1}?

[Michaelvll]

Thanks for the update @MaoZiming! I tried the following, but an error occurred:

resources:
  any_of:
    - cloud: aws
      region: us-east-1
      accelerators: A100:8
    - cloud: gcp
      accelerators: T4:4
    - cloud: aws

run:
  echo hi

sky launch -c test test.yaml twice:

sky/backends/cloud_vm_ray_backend.py", line 4347, in _check_existing_cluster
    assert len(task.resources) == 1
AssertionError

[MaoZiming]

@Michaelvll Thanks! pushed a fix

[Michaelvll]

Thanks for the great effort @MaoZiming! The LGTM now. Once it has passed the smoke test and any additional manual tests, it should be good to go in. : )

[MaoZiming]

Thanks for the review! Addressed the comments.
The smoke tests should be good (except the ones I mentioned in the issue)