ipfs ls API commands hang while ipfs object ls and gateway listing works

[dokterbob]

Version/Platform/Processor information (from ipfs version --all):
0.4.3rc3 and 0.4.2

Type (bug, feature, meta, test failure, question): bug
Area (api, commands, daemon, fuse, etc): commands
Priority (from P0: functioning, to P4: operations on fire): P1

Description:
For a lot of hashes, executing ipfs ls (or ipfs file ls) gets stuck (or is at least tediously slow) while ipfs files ls (after cp'ing the file into MFS) and ipfs objects ls works normally. When ipfs files ls -l is executed, however, the same lag occurs.

My suspicion is that this is due to the way file sizes are calculated, which is different from object sizes. However, given that object sizes seems ok for the gateway interface, perhaps we should normalize to this and create and make calculating the 'real' file sizes optional.

Related to #3056.

[whyrusleeping]

@dokterbob Yeah, thats because ipfs ls by default will resolve what type each directory entry is. You can use the --resolve-type=false option to disable that.

[dokterbob]

It's interesting - and potentially problematic - to find that resolving the type of a resource takes so long. As far as I know, the first block of the particular resource needs to be fetched in order to look at its protobuf to resolve the type.

However, it seems that in some cases where the linked-to resource is actually available, resolving the type still takes a very long time. I have yet to reliably verify this as it is hard to bring IPFS back into a 'clean' state.

This makes me wonder: is ipfs ls somehow doing a recursive lookup?

If not, would it perhaps make sense to serialize object metadata out to containing directories, similar to object size? This would also entail the full mimetype, allowing for a much faster, more pretty rendering of indexes and seems to make proper FS integration a lot more feasable.

[whyrusleeping]

@dokterbob Yeah, thats definitely being considered when we move to ipld. We will be able to more easily keep extra information in the directory object to support this.

It definitely is kinda weird that ipfs ls is taking considerably longer than just accesing the block. although I have seen this quite commonly where someone adds a directory, and then views the directory through the gateways (caching that directory node) and then takes their node offline. When that happens, only the root directory node is available to anyone, causing ipfs ls to hang.

[dokterbob]

It seems a bit worse than what you suggested. Even for available resources, listing them with type resolving takes forever(-ish) for large directories. Executing ipfs ls --resolve-type=false on a directory with a few hundred subdirectories (with the files available on another, connected, host) takes less than a second. Executing the same with resolve-type=true seems to, basically, hang.

[aschmahmann]

You can quickly resolve the directories by using --stream=true --size=false --resolve-type=false

[helonaut]

I would like to re-open this issue as we keep on running into this on a Mobian platform (ARM64).
Is there any way to specify these arguments as default for the ipfs ls command through the config file?
If so, how please? If not, is there a way to make this possible? If not, is it possible to change the default behavior of the command?