Multipart upload for large data.

[khustup2]

🚀 🚀 Pull Request

Use upload_fileobj to upload large data into s3.

Impact

• Bug fix (non-breaking change which fixes expected existing functionality)
• Enhancement/New feature (adds functionality without impacting existing logic)
• Breaking change (fix or feature that would cause existing functionality to change)

Description

Things to be aware of

Things to worry about

Additional Context

[codecov]

Codecov Report

Attention: 2 lines in your changes are missing coverage. Please review.

Comparison is base (133e0a4) 83.34% compared to head (350e671) 83.92%.

Files	Patch %	Lines
deeplake/core/storage/s3.py	66.66%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2734      +/-   ##
==========================================
+ Coverage   83.34%   83.92%   +0.58%     
==========================================
  Files         233      233              
  Lines       26447    26452       +5     
==========================================
+ Hits        22041    22200     +159     
+ Misses       4406     4252     -154     

Flag	Coverage Δ
unittests	83.92% <66.66%> (+0.58%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[sonarcloud]

Quality Gate passed

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
60.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

[nvoxland-al]

Is put_object enough faster than upload_fileobj that it's worth having the else clause still? Or can we always use upload_fileobj ?

[khustup2]

put_object is direct mapping to the PutObject api call and considered as a default way to upload object. I'm not sure about the performance differences, but I see upload_fileobj as an edge case scenario and not as a default upload method.