Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update s5cmd to 2.1.0 to resolve memory issue during sync #16

Merged
merged 1 commit into from
Jul 26, 2023
Merged

Update s5cmd to 2.1.0 to resolve memory issue during sync #16

merged 1 commit into from
Jul 26, 2023

Conversation

dcarrion87
Copy link
Contributor

@dcarrion87 dcarrion87 commented Jul 26, 2023 •
edited
Loading

Related Tasks

What

  • s5cmd loads too much during sync. Switch to version where issue is resolved.

Note new features and breaking changes in Slack channels after release. https://github.com/peak/s5cmd/releases/tag/v2.1.0

Why

  • OOMs are bad.

@dcarrion87 dcarrion87 linked an issue Jul 26, 2023 that may be closed by this pull request
s5cmd out of memory issue #15
Closed
@dcarrion87 dcarrion87 requested a review from a team July 26, 2023 05:20
@dcarrion87 dcarrion87 self-assigned this Jul 26, 2023
@dcarrion87 dcarrion87 added infra bug Something isn't working labels Jul 26, 2023
@rfk
Copy link

rfk commented Jul 26, 2023

I manually updated to this version of s5cmd and can confirm that it makes a big difference on large data syncs:

Capture

guillaume-troadec
guillaume-troadec approved these changes Jul 26, 2023
View reviewed changes
Copy link
Contributor

@guillaume-troadec guillaume-troadec left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@dcarrion87 dcarrion87 merged commit 03d9bf2 into main Jul 26, 2023
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@guillaume-troadec guillaume-troadec guillaume-troadec approved these changes

Assignees

@dcarrion87 dcarrion87

Labels
bug Something isn't working infra
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

s5cmd out of memory issue
3 participants
@dcarrion87 @rfk @guillaume-troadec