New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix(artifacts): add s3 multipart uploading for artifact files #5377
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we need to add tests before I'm willing to merge this. I know it's a pain to weave it in, but I think we also need to rework it a bit to enable reading only a chunk of a file at a time; ideally in a way that won't be too hard to adapt to asyncio / aiofiles in the future.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for adding all the new tests! I know this was a long process, but this looks good to me 👍
Fixes
Fixes WB-11826
Description
Enables multipart upload for S3 for artifact files. In the past, artifact files larger than 5 GiB would get errors when uploading to AWS S3. Now using S3 multipart upload we can upload files up to 5 TiB if internet speed and disk space permits.
Testing
🤖 Generated by Copilot at 8268f45