This repository has been archived by the owner on Mar 17, 2021. It is now read-only.
Improve upload process with async just-in-time process #49
Labels
enhancement
New feature or request
Right now our upload process works, but it will be inefficient for large data.
Below is our current process:
This will slow down the upload process with large file since sharding takes longer time for big file.
A more efficient way will be using "async" way.
Step 1 and 2 will remain unchanged
3. Once the first shard finishes, we can distribute it first to the network when client is still writing the next shard
4. There will be very small chance that uploading the first shard will be faster than client writing the 2nd shard, but if it really happens, we can use if else statement to check if the 2nd shard completes yet before implementing distribution
The text was updated successfully, but these errors were encountered: