Adding large directory (10k items) results in an error

[hsanjuan]

A user reported that adding a directory with 10k items on cluster does not work well.

[hsanjuan]

Indeed adding a directory with many files hits a dagservice: block not found error.

[hsanjuan]

So the latest versions of go-unixfs that we are use have a DynamicDirectory type which switches itself from BasicDirectory to HAMTDirectory (and back) depending on how many children have been added. When that switch happens, it reads all the children from the BasicDirectory to add them to the HAMTDirectory as new links.

I believe we can patch go-unixfs to not re-read children and just re-use links in the BasicDirectory to add them to the HAMT directory.

[hsanjuan]

Basic testing suggests the problem is solved with ipfs/go-unixfs#120.

[hsanjuan]

The problem from the cluster side is that our DAGService is write-only and we cannot really read blocks that have written previously, since those blocks may have been written in the IPFS daemon of a completely different cluster peer. The assumptions for Adding is that we just write blocks in places, no need to re-read them.