YAML support for sky.Storage

[romilbhardwaj]

Sky storage (#63) will also require YAML support so Storage objects can be defined as a part of the YAML schema. Maybe something like:

name: resnet-app
workdir: ~/Downloads/tpu

resources:
  cloud: aws
  instance_type: p3.2xlarge

storage:
  name: imagenet-bucket
  source_path=s3://imagenet-bucket
  mount_path=/imagenet/

[michaelzhiluo]

@romilbhardwaj @concretevitamin Since mount_path is not an argument for Storage anymore, what are your 1) suggestions for putting mount_path in the config and 2) how to scale the yaml for multiple Storage objets?

[romilbhardwaj]

Good point - we could take the kubernetes approach and define storage objects as a list of storages each with a unique id (e.g. see volumes in k8s Pods), and then specify this id in storage_mounts field like so:

name: resnet-app
workdir: ~/Downloads/tpu

resources:
  cloud: aws
  instance_type: p3.2xlarge

storage:
  -name: imagenet-bucket
   source_path=s3://imagenet-bucket
  -name: mscoco-bucket
   source_path=s3://mscoco-bucket


storage_mounts:
  -storage: imagenet-bucket
   mount: /imagenet/ 
  -storage: mscoco-bucket
   mount: /mscoco/

If want to stay true to our python API, then we can define storage objects inline in storage_mounts:

storage_mounts:
  -storage:
      -name: imagenet-bucket
      -source_path=s3://imagenet-bucket
   mount: /imagenet/

What do you guys think?

[concretevitamin]

How should we express "get_or_copy_to_s3()"? A direct attempt:

storage_mounts:
  - storage:
      - name: imagenet-bucket
      - source_path: /local/data/imagenet
      - get_or_copy_to_s3: true
      - mount: /imagenet

[michaelzhiluo]

As we discussed today, here's the schema:

name: resnet-app
workdir: ~/Downloads/tpu

resources:
  cloud: aws
  instance_type: p3.2xlarge

storage:
  -name: imagenet-bucket
   source=~/imagenet/
   force_upload: [s3, gcs] # Could be [s3] or [gcs], default: None
   persistent=True

storage_mounts: {imagenet-bucket: /myimagenet/}

[romilbhardwaj]

Evolving from our discussion today, here's the final schema I propose:

storage:
  -name: imagenet-bucket
   source: ~/imagenet/
   force_backend: [s3, gcs] # Could be [s3] or [gcs], default: None
   persistent: True

storage_mounts:
  -storage: imagenet-bucket
   mount_path: /imagenet/ 

1. Thinking a little more about it, I propose we change force_upload to force_backend: [s3, gcs, blob]. The reason we changed from StorageBackend to AbstractStore in our code was because a "backend" class implied a singleton class to access a storage service, but we were instantiating multiple instances of the "backend" which didn't make semantic sense. Here, however, "backend" seems appropriate because it is indeed selecting a backing service that provides object storage.

2. I would keep the key persistent (instead of persist_on_cloud) since that's the only notion of persistence (files can persist or not persist only on the cloud, local persistence is under the user's control)

3. storage_mounts: {imagenet-bucket: /myimagenet/} still feels a little iffy to me. I feel keys in a YAML should indicate a property name instead of a value. Borrowing ideas from k8s, our storage_mounts should be like:

storage_mounts:
  -storage: imagenet-bucket
   mount_path: /imagenet/ 

I agree, this adds an additional line but allows us to have a well defined and consistent schema. My opinion on this is influenced by k8s, so please feel free to push back if you feel this is too much boilerplate code to write.

4. I see sky.Storage as a lower level implementation of file_mounts. We can (and should) replace the internals of file_mounts to use (sky.Storage) instead of Ray autoscaler/rsync. For instance, behind the scenes we should be able to convert a YAML withfile_mounts: {/imagenet/: /home/romilb/imagenet/} to a Storage object + storage_mount in the task.

[concretevitamin]

This has landed by @michaelzhiluo #121.