Sky Data

[michaelzhiluo]

Completed Items

• Backends for S3 and GCS Blobs
• Creating S3/GCS Blobs
• Deleting S3/GCS Blobs
• Uploading Data (parallel upload) to S3/GCS Blobs
• Downloading Data (not parallel yet) from S3/GCS Blobs
• Transfer between S3 and GCS Blobs (1-2 GB/s bandwidth for Imagenet, 2-3 min transfer time)
• Moving AWS authentication across all nodes
• Integrating Storage Classes and Backend into Sky (currently it is a standalone module)
• Download S3/GCS data onto the Cloud VM

TODOs

• [Optional] Mounting S3/GCS Blob onto all Cluster nodes (Optional, to @concretevitamin 's suggestion )

[michaelzhiluo]

@romilbhardwaj @concretevitamin Ready to review!

[romilbhardwaj]

Fantastic work! Left some comments, but I generally like how this has come out to be

[romilbhardwaj]

As we add more backends, we'll probably have to add more logic here for cross-cloud transfers.. In a future PR we should clean this up by adding something like a transfer registry which maps Tuple[Backend1, Backen2] -> TransferFn(Backend1, Backend2)

[concretevitamin]

Looking great!! First pass, didn't look into all details but I tried to raise pressing high-level concerns/suggestions.

---

Question:

For GNN, I imagine they will always write

s = sky.Storage('my-data', source='/data/gnn/train')
# First run: upload.
# Second+ runs: sync (some overhead even if no change to data).
# This overhead is OK for now; future we can optimize it away.
s.to_s3()
task.set_storage_mounts({s: '/data/gnn/train'})

Is that right?

[Not for this PR] Suppose they now run this Sky script in a different machine without access to source='/data/gnn/train'. We need a way (sky status? sky storage?) to enable them to find what S3 path is backing my-data. So that they can write sky.Storage('my-data', source=<correct s3 path>) instead.

[romilbhardwaj]

Also, I just realized we will need to add support for Storage in our YAML schema. That's probably best done as a separate PR...
cc @concretevitamin @michaelzhiluo @gmittal

[michaelzhiluo]

Also, I just realized we will need to add support for Storage in our YAML schema. That's probably best done as a separate PR... cc @concretevitamin @michaelzhiluo @gmittal

Will be done in a followup PR after this is merged

[michaelzhiluo]

@romilbhardwaj @concretevitamin Think its good to go! Addressed 99% of the comments.

[concretevitamin]

Looks good! Mostly some more API ideas. Should be good to go after that.

[romilbhardwaj]

Fantastic work! Left a minor comment.

Also, in a future PR we should investigate directly mounting the storage using GcsFuse/s3Fuse instead of copying to local EBS:

1. Reduces setup time and EBS size (at the cost of increased disk latency/xput, need to evaluate this tradeoff)
2. Allows for writes to Storage which users can access from other workers/local dev machine

[romilbhardwaj]

Note to team: As sky.Storage becomes the standard method of uploading/interfacing with data in Sky, we should deprecate CloudStorage and move it's functionality to AbstractStore
cc @concretevitamin @michaelzhiluo

[michaelzhiluo]

Agreed, we can refactor in a later PR @romilbhardwaj @concretevitamin

[concretevitamin]

Outstanding work @michaelzhiluo !

[michaelzhiluo]

Thanks for the reviews! @romilbhardwaj @concretevitamin