Skip to content
This repository has been archived by the owner on Mar 17, 2021. It is now read-only.

Improve upload process with async just-in-time process #49

Closed
floalex opened this issue Apr 5, 2018 · 1 comment
Closed

Improve upload process with async just-in-time process #49

floalex opened this issue Apr 5, 2018 · 1 comment
Labels
enhancement New feature or request

Comments

@floalex
Copy link
Member

floalex commented Apr 5, 2018

Right now our upload process works, but it will be inefficient for large data.

Below is our current process:

  1. Encrypt the file
  2. Once encryption finishes, divide it in to k number of shards
  3. Once all the shards finishes, distribute it to the network

This will slow down the upload process with large file since sharding takes longer time for big file.

A more efficient way will be using "async" way.
Step 1 and 2 will remain unchanged
3. Once the first shard finishes, we can distribute it first to the network when client is still writing the next shard
4. There will be very small chance that uploading the first shard will be faster than client writing the 2nd shard, but if it really happens, we can use if else statement to check if the 2nd shard completes yet before implementing distribution

@floalex floalex added the enhancement New feature or request label Apr 6, 2018
@WilfredTA
Copy link
Member

Done in the latest commit

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants