Fast removal of S3 Storage buckets with 10k-1 million objects #893

michaelzhiluo · 2022-06-02T21:32:16Z

Fixes #434.

Test

touch ~/Downloads/hallo/{00001..10000}.c

Sky YAML

file_mounts:
  /checkpoints:
    name: michaels-new-bucket-1
    source: ~/Downloads/hallo
    store: s3
    mode: MOUNT

sky launch -c hello storage.yaml , sky storage delete michaels-new-bucket-1

Before: 3-4 min
After: 10-20(s)

* Update the installation and tutorial * fix * Add description in README * fix * Add link to examples * Polish readme * Update readme * readme * Fix comment

* gather adaptors * give instructions

* Limit grpcio version * rename disk_size * format

* restore README to the original place

* Support 'name:cnt' accelerators spec in YAML * Fixes #373: 'sky start/down' should error out

Reasons: - `K80:4` is not available on AWS, arguably the most common cloud our target users use (so they will hit resource unavailable) - `V100:1` is available on all three clouds and is a popular GPU

* Ok * Addressed all comments * Changed to new git link * ok

* Fix check_local_gpus * Break a line to meet 80 char constraint * Address the review comments

* Initial Draft * Delete bad file * ? * ??? * format * sky storage status; sky storage down * Fixed * Done * Addressed Comments * Addressed Comments, TODO: Documentation * Documentation added * ok

* Initial Draft * Delete bad file * ? * ??? * format * sky storage status; sky storage down * Fixed * Done * Addressed Comments * Addressed Comments, TODO: Documentation * Documentation added * ok * Addressed Zhanghao's comments * Fix * SGTM

* wip: Add setup in provision pipeline * Fix gcp/azure * remove useless variables * minor fix * Add some TODOs * Fix comments * Fix comments * Fix gcp/azure initialization_commands * Remove setup from template * Fix setup directory * Change rsync back to -Pavz * Remove unused argument * fix file_mount/dir_mount

* WIP Debug * revert file_mounts using storage mounts and update docs * remove print * Fix credential cmd * lint Co-authored-by: Zongheng Yang <zongheng.y@gmail.com>

… status` (#427) * A quick fix in killing sky docker containers * Add a comment

…422) * Add conda activate support to bashrc * Add doc and make sure conda activate works * bring back conda activate command for GCP * Move comment to quickstart * format * Fix comments * Add test/example of using user_script * Fix indents * bash -i only for conda activate * Fix the SKY_NODE_IPS fail to pass to the shell script * Update readme * update env_check * Fix comments * Change to head -n1

* Remove -i option * Fix docs * Fix run.sh * add comments * fix comment * format

* Fix az key generation * Add private keys to retval

* ok * Shorten YAML * Ok * Done * Nit added * Romil's changes

* Azure import * Add docs.

…/...` not `~/sky_workdir/workdir/...`) (#443) * Fix * Resnet Example working * Fix

* Fix gitignore in file_mounts * Add test * Update j2 for filter * format

* Remove tracebacks * Fix job fail color * fix comments * Hide tracebacks * Fix #442 * fix `workdir` becomes `~/sky_workdir/workdir` #442 * add logging error for job_id problem * format * update error message for retry * Update docs * Fix login * Add more checks * format * fix return type * format * refactor returncode handling * Update return handling * Fix filemount testing

* Switch to ray job logs * Optimize log tailing * format * Fix exec logging * Add comment * Move back to run_with_log * Bring back our tailing function for progress bar * format * Fix comments * Remove check argument from run_with_log * Add comment * Add comment * lint

#862) * Distinguish controller failure and user failure * Add hints for getting error messages * Fix * update message * rename to cluster failure * message for cluster failed as well * Fix failing * address comments * Add id for end of logs * Split resource failure and controller failure * Fix terminal state * Address comments * fix typo

* Add some docs * update * fix * fix * update * address comments * reorg * reorg and add fig * Add imgs * fix * update

* Distinguish controller failure and user failure * Add hints for getting error messages * Fix * update message * rename to cluster failure * message for cluster failed as well * Fix failing * Add pending state for spot jobs * Fix job id * format * address comments * Add id for end of logs * fix pending * Add name and resources * format * Add failed status check for spot state * Refactor the backend interface * address comments * fix status * address comment * Fix comment

romilbhardwaj

Awesome! Thanks for finding this fix and implementing it! Some comments, similar to #888.

sky/data/storage.py

sky/exceptions.py

…into storage_fast_delete

michaelzhiluo · 2022-06-06T08:00:19Z

Thanks for the review! @romilbhardwaj If you have time, PTAL

…_fast_delete

romilbhardwaj

LGTM! Thanks for adding this! Two small typos in comments

sky/data/storage.py

Michaelvll and others added 30 commits February 17, 2022 23:39

Fixes for document and README (#360)

61020db

* Update the installation and tutorial * fix * Add description in README * fix * Add link to examples * Polish readme * Update readme * readme * Fix comment

Update links in documentation and README (#366)

f212f1e

Improve adaptors (#359)

62fae46

* gather adaptors * give instructions

Limit grpcio version for #369 (#370)

849e753

* Limit grpcio version * rename disk_size * format

Remove typo from README (#371)

232fc65

Fix Azure credential errors on AWS (#372)

e26ef9c

fix typo (#376)

fd57445

Restore README to the original place (#374)

47e264d

* restore README to the original place

Support 'name:cnt' accelerators spec in YAML (#396)

edb7f10

* Support 'name:cnt' accelerators spec in YAML * Fixes #373: 'sky start/down' should error out

README example: change K80:4 to V100:1 (#399)

99344b1

Reasons: - `K80:4` is not available on AWS, arguably the most common cloud our target users use (so they will hit resource unavailable) - `V100:1` is available on all three clouds and is a popular GPU

Workdir Docs (#405)

6276b0f

* Ok * Addressed all comments * Changed to new git link * ok

Fix check_local_gpus (#411)

4486851

* Fix check_local_gpus * Break a line to meet 80 char constraint * Address the review comments

Sky Storage CLI (#338)

8d17229

* Initial Draft * Delete bad file * ? * ??? * format * sky storage status; sky storage down * Fixed * Done * Addressed Comments * Addressed Comments, TODO: Documentation * Documentation added * ok

Sky Storage CLI - Polishing (#414)

d19a6fd

* Initial Draft * Delete bad file * ? * ??? * format * sky storage status; sky storage down * Fixed * Done * Addressed Comments * Addressed Comments, TODO: Documentation * Documentation added * ok * Addressed Zhanghao's comments * Fix * SGTM

Minor fix (#419)

3f43c99

Revert to using file_mounts for object mounting (#412)

2d033aa

* WIP Debug * revert file_mounts using storage mounts and update docs * remove print * Fix credential cmd * lint Co-authored-by: Zongheng Yang <zongheng.y@gmail.com>

[docker] Quick fix to remove already-dead docker containers from `sky…

24a49c8

… status` (#427) * A quick fix in killing sky docker containers * Add a comment

Update documentation with provisioning quickstart (#415)

63eff9e

Add improved progress meters + log suppression (#425)

a8141ab

Add cross-cloud failover to docs. (#433)

d71e2d2

Remove -i for bash user_script.sh (#436)

f7df7a8

* Remove -i option * Fix docs * Fix run.sh * add comments * fix comment * format

Fix Azure key generation (#429)

b45f04c

* Fix az key generation * Add private keys to retval

[Storage/File-Mounting] Fix Symlink Issue + Fix File Mounting (#431)

b1d7431

* ok * Shorten YAML * Ok * Done * Nit added * Romil's changes

Fix error message when azure-cli is not installed (#424)

e5b5847

* Azure import * Add docs.

Fixing Relative Directory for Workdir and File Mounts (`~/sky_workdir…

bfd93f9

…/...` not `~/sky_workdir/workdir/...`) (#443) * Fix * Resnet Example working * Fix

Fix gitignore for rsync (#448)

93e036e

* Fix gitignore in file_mounts * Add test * Update j2 for filter * format

concretevitamin and others added 10 commits June 1, 2022 11:04

Minor: reformat sky show-gpus output. (#877)

9825303

Add YAML schema check (#680)

5b81718

Add explanations on spot docs (#852)

06f16b4

* Add some docs * update * fix * fix * update * address comments * reorg * reorg and add fig * Add imgs * fix * update

Fix typo (#880)

a53e013

Make sky launch prompting consistent with interactive nodes (#867)

f3c865e

remove azureTokens.json from the credential list (#883)

b5040ae

Fast removal of buckets

518cfa2

Fast removal of buckets

52f01e5

michaelzhiluo changed the title ~~Fast removal of Storage buckets with 10k-1 million objects~~ Fast removal of S3 Storage buckets with 10k-1 million objects Jun 2, 2022

michaelzhiluo requested review from romilbhardwaj and concretevitamin June 2, 2022 21:32

michaelzhiluo added 4 commits June 2, 2022 16:10

Replace os.system with subprocess

6c4d32b

Replace os.system with subprocess

b1ed515

fix syntax

2649f5e

fix syntax

03dd2af

romilbhardwaj reviewed Jun 4, 2022

View reviewed changes

sky/data/storage.py Outdated Show resolved Hide resolved

sky/data/storage.py Outdated Show resolved Hide resolved

sky/data/storage.py Outdated Show resolved Hide resolved

sky/exceptions.py Outdated Show resolved Hide resolved

suquark force-pushed the storage_fast_delete branch from 03dd2af to 2649f5e Compare June 5, 2022 11:09

suquark force-pushed the master branch from 75f2804 to 83dcc4b Compare June 5, 2022 18:15

michaelzhiluo added 2 commits June 6, 2022 00:45

Merge branch 'storage_fast_delete' of https://github.com/sky-proj/sky …

bad31d5

…into storage_fast_delete

Addressed Romil's comments

78e656e

Merge branch 'master' of https://github.com/sky-proj/sky into storage…

ccc32c6

…_fast_delete

romilbhardwaj approved these changes Jun 6, 2022

View reviewed changes

sky/data/storage.py Outdated Show resolved Hide resolved

sky/data/storage.py Outdated Show resolved Hide resolved

michaelzhiluo added 2 commits June 6, 2022 02:03

fix comment

f08c7bd

Fix comments 2

9b86bdf

michaelzhiluo merged commit 6ac9687 into master Jun 6, 2022

suquark pushed a commit that referenced this pull request Jun 7, 2022

Fast removal of S3 Storage buckets with 10k-1 million objects (#893)

3739734

suquark pushed a commit that referenced this pull request Jun 7, 2022

Fast removal of S3 Storage buckets with 10k-1 million objects (#893)

f69cfb4

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fast removal of S3 Storage buckets with 10k-1 million objects #893

Fast removal of S3 Storage buckets with 10k-1 million objects #893

michaelzhiluo commented Jun 2, 2022 •

edited

Loading

romilbhardwaj left a comment

michaelzhiluo commented Jun 6, 2022

romilbhardwaj left a comment

Fast removal of S3 Storage buckets with 10k-1 million objects #893

Fast removal of S3 Storage buckets with 10k-1 million objects #893

Conversation

michaelzhiluo commented Jun 2, 2022 • edited Loading

Test

romilbhardwaj left a comment

Choose a reason for hiding this comment

michaelzhiluo commented Jun 6, 2022

romilbhardwaj left a comment

Choose a reason for hiding this comment

michaelzhiluo commented Jun 2, 2022 •

edited

Loading