Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(artifacts): add s3 multipart uploading for artifact files #5377

Merged
merged 35 commits into from May 10, 2023

Conversation

estellazx
Copy link
Contributor

@estellazx estellazx commented Apr 19, 2023

Fixes

Fixes WB-11826

Description

Enables multipart upload for S3 for artifact files. In the past, artifact files larger than 5 GiB would get errors when uploading to AWS S3. Now using S3 multipart upload we can upload files up to 5 TiB if internet speed and disk space permits.

Testing

  • Manually testing S3 uploads on local and QA. Existing GCS uploads are not effected and S3 is able to upload files over 5 GB now. Was able to upload 200 GB under fast wifi without any problems
  • Manually backwards compatibility test with https://multi-part.wandb.ml/ which is set to (0.30.0 before multipart server changes in 0.32.0)
  • Unit tests

🤖 Generated by Copilot at 8268f45

Oh we are the wandb crew and we have a job to do
We're uploading artifacts in chunks to S3
We use the Api class and the WandbStoragePolicy
And we heave away on the count of three

@estellazx estellazx requested a review from a team as a code owner April 19, 2023 16:24
@github-actions github-actions bot added cc-fix and removed cc-fix labels Apr 19, 2023
Copy link
Contributor

@moredatarequired moredatarequired left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need to add tests before I'm willing to merge this. I know it's a pain to weave it in, but I think we also need to rework it a bit to enable reading only a chunk of a file at a time; ideally in a way that won't be too hard to adapt to asyncio / aiofiles in the future.

wandb/sdk/internal/internal_api.py Outdated Show resolved Hide resolved
wandb/sdk/internal/internal_api.py Show resolved Hide resolved
wandb/sdk/internal/internal_api.py Outdated Show resolved Hide resolved
wandb/sdk/internal/internal_api.py Show resolved Hide resolved
wandb/sdk/wandb_artifacts.py Show resolved Hide resolved
wandb/sdk/wandb_artifacts.py Outdated Show resolved Hide resolved
wandb/sdk/wandb_artifacts.py Outdated Show resolved Hide resolved
@github-actions github-actions bot added cc-fix and removed cc-fix labels Apr 25, 2023
Copy link
Contributor

@moredatarequired moredatarequired left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding all the new tests! I know this was a long process, but this looks good to me 👍

@github-actions github-actions bot added cc-fix and removed cc-fix labels May 10, 2023
@raubitsj raubitsj merged commit 2028e00 into main May 10, 2023
52 checks passed
@raubitsj raubitsj deleted the estella/multipart branch May 10, 2023 14:39
@dmitryduev dmitryduev added this to the sdk-2023-06.1 milestone May 17, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants