Increase write speed by parallelizing writing to the backing store before acquiring a lock on set

[TheGreatAlgo]

Possible Ways to Improve

Move the blocking to after the store.save_raw to limit blocking

Current: Get lock -> write 2mb -> built hamt -> unlock -> get lock -> write 2mb -> built hamt.

Improvement -> parallel write 20mb -> get lock on 2mb chunk -> built hamt -> unlock -> second portion is immediately available and no longer required to wait to write 2mb block

Second Improvement Idea:

Use memory store on the hamt kv pairs. This allows parallel writing to IPFS but limiting the blocking since when dealing with memory its much quicker. Use the IPFS hashing to hash the data and properly built the hamts in memory.

After its done, just write the content of the hamts and it the hashes should align.

[abidsikder]

Move the blocking to after the store.save_raw to limit blocking

Current: Get lock -> write 2mb -> built hamt -> unlock -> get lock -> write 2mb -> built hamt.

Improvement -> parallel write 20mb -> get lock on 2mb chunk -> built hamt -> unlock -> second portion is immediately available and no longer required to wait to write 2mb block

This sounds promising, and the code change itself should be simple. The hard part will be writing a multithreaded test to ensure that a speedup exists, and that should be the first step; the code change seems fairly simple, which is a good thing.

Second Improvement Idea:

Use memory store on the hamt kv pairs. This allows parallel writing to IPFS but limiting the blocking since when dealing with memory its much quicker. Use the IPFS hashing to hash the data and properly built the hamts in memory.

After its done, just write the content of the hamts and it the hashes should align.

In general, py-hamt will avoid keeping a buffer that will need to then be flushed to the backing store, as this will add complexity in the API. HAMTs will need to know when they need to flush, and failures in the middle of using the HAMT might mean that data that was in progress has been lost. Internally, this will also mean there will be added code to look at the buffer before going to the tree currently in the Store.

Future PRs that address this issue should address the first enhancement idea.