Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: BREAKING CHANGE: deletion logic by each page and removing the progress bar and -q option #168

Merged
merged 23 commits into from
Apr 15, 2024

Conversation

go-to-k
Copy link
Owner

@go-to-k go-to-k commented Apr 13, 2024

Closes #164.

This PR includes the following:

  • Add the ListObjectVersionsByPage method in the S3 client.
    • Using next keys.
  • Change the deletion logic in the S3 wrapper.
    • Delete by each page getting from the ListObjectVersionsByPage operation.
    • This has reduced the time required for deletion to about one-half of the previous time.
      • against a bucket with 100,000 objects created by the test deploy shell
      • deletion time
        • latest version(v0.15.0)
          • 2m50.029s
        • this PR
          • 1m30.138s
    • Memory utilization has also improved significantly (it used to increase in proportion to the number of objects, but now it is almost constant).
  • Remove the progress bar.
    • However, the number of deleted objects is displayed live.
  • Remove the quiet (-q) option.
    • I've decided that I no longer need this option since I've removed the progress bar.
    • This is a BREAKING CHANGE.

cls3-page-speed

@go-to-k go-to-k marked this pull request as ready for review April 13, 2024 17:54
@go-to-k
Copy link
Owner Author

go-to-k commented Apr 13, 2024

I'll release it once I've verified a few things are working. I would also like to check memory usage before doing so.

@go-to-k go-to-k changed the title feat: BREAKING CHANGE: deletion logic by page and removing the progress bar and -q option feat: BREAKING CHANGE: deletion logic by each page and removing the progress bar and -q option Apr 14, 2024
@go-to-k go-to-k merged commit 03444f4 into main Apr 15, 2024
2 checks passed
@go-to-k go-to-k deleted the list-by-page branch April 15, 2024 09:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature] faster deletion : on the fly deletion of each page
1 participant