It can only submarine directories of up to 5697 files

[clrke]

Web UI when the submarined directory contains 5698 files:

Web UI when the submarined directory contains 5697 files:

Our project is an NFT collection of 10,000. Right now we're forced to split the ipfs files into two because of this.

[kf106]

I have fixed a number of bugs I found in the scripts, mainly to do with wrong paths being included (I am assuming you worked around this).

After that, I tried generating a test folder with 6000 entries, and I get a 404 not found error.

Subsequently I generated a test folder with 5697 entries, and get a listing of all the file names.

So I have reproduced the problem.

My initial suspicion here is that this is a bug in the ipfs web ui. I will investigate further.

[kf106]

It looks like it is an IPFS limitation:

kf106@media-pc-3:~/GIT/ipfs-submarine$ IPFS_PATH=$HERE/ipfs-public ipfs file ls bafybeiatiioxsuokbezw5plj6xsstckrqpmmxn3h6eby5sq6xjjji3oiwq
Error: cannot list large directories yet

I'm now investigating something called "directory sharding", which is experimental, but is meant to allow larger directories.

[kf106]

Nope, nothing in that topic area provides a solution. I think it's an IPFS problem, and that's something I can't fix.

[kneelsdev]

@kf106 I figured out a fix. See this ipfs pr: https://github.com/ipfs/kubo/pull/8527/files

# Test automatic sharding and unsharding

# We shard based on size with a threshold of 256 KiB (see config file docs)
# above which directories are sharded.
#
# The directory size is estimated as the size of each link. Links are roughly
# the entry name + the CID byte length (e.g. 34 bytes for a CIDv0). So for
# entries of length 10 we need 256 KiB / (34 + 10) ~ 6000 entries in the
# directory to trigger sharding.

Increasing the sharding threshold will let you submarine larger directories. I just set it to 1GB.
"Internal": {
"UnixFSShardingSizeThreshold": "1GB"
},