files.cat and object.links / object.data fail on some files they should work on

[mitra42]

• Version: 0.28.1
• Platform: OSX 10.13.3, Node v8.8.1
• Subsystem: files.cat and object.links/object.data

Type: Bug

Severity: High

Description: files.cat or object.links, object.data dont work on some links they almost certainly should do.

Steps to reproduce the error:

See the attached file which can be run with

mv test_ipfs_video.js.txt test_ipfs_video.js   # Because git wont let me attach a .js
[test_ipfs_video.js.txt](https://github.com/ipfs/js-ipfs/files/1800152/test_ipfs_video.js.txt)

node test_ipfs_video.js

it tests retrieving two different files via files.cat and via the object.links, object.data sequence used in 8f8c539
One file works, the other fails.

The differences / similarities.
Both are retrievable via the gateway
PDF https://ipfs.io/ipfs/Qmbzs7jhkBZuVixhnM3J3QhMrL6bcAoSYiRPZrdoX3DhzB
VIDEO https://ipfs.io/ipfs/zdj7Wc9BBA2kar84oo8S6VotYc9PySAnmc8ji6kzKAFjqMxHS
Both are multi-block, the first has 2 links, the second 46
The PDF was uploaded via http with /api/v0/add maybe 6 months ago
The VIDEO was uploaded via http with /api/v0/urlstore about 2 months ago

The PDF is retrieved successfully with either files.cat or object.links / object.data (see the code for what I mean by this)

On the VIDEO
files.cat fails with Failed to unmarshal node
object.data returns undefined on the first block of the video

I don't think the difference is PDF v VIDEO since its not accuracy of the data that is the issue, its the failure to retrieve it at all with files.cat

[daviddias]

@mitra42 are you 100% sure that the file was added with files.add? Looks like it is trying to unmarshall non unixfs protobuf. Thanks for the repro case :)

[mitra42]

Not with files.add, I'm 99% sure the failing file was added with urlstore as that was the switch from Q* to z* hashes, I don't know how the successful one was added (could have been files.add or dag.put).

To be certain I tried two more files, one added a couple of days ago and the other I just added. Both were added via the http api to urlstore. The weird thing is that one fails in the object.data (retrieves undefined) and the other fails in unmarshal (returns undefined). All of these files except the PDF fail in files.cat

See attached - same instructions, rename to test_ipfs_video.js and run node test_ipfs_video.js
test_ipfs_video.js.txt

[kyledrake]

This I believe is related to the urlstore code not currently renewing DHT provides automaticaly, so we're working on adding this functionality through a different process for the root CIDs.

[autonome]

@mitra42 is this is still a problem with contemporary IPFS?

[mitra42]

Hi - sorry, I don't know, This was over 2 years ago, so my memory is fuzzy and I haven't touched IPFS code for a while (its been turned off in dweb.archive.org for a long time by default because of other bugs/issues)

My vague memory is that Kyle was probably correct (a DHT scaling problem with IPFS which AFAIK was never fixed, and was part of why we dropped trying to keep the IPFS gateway alive).

I do see workarounds for odd problems in https://github.com/internetarchive/dweb-transports/blob/master/TransportIPFS.js#L334 and that might have been related.

Sorry I can't be of more help.

[autonome]

Thanks @mitra42.

Triaged - need someone to update the testcase, and re-run.