ipfs.get is not working with sharded directories

[daviddias]

• Version: 0.28.2
• Platform: Mac OS X
• Subsystem: unixfs-engine

Type: Bug

Severity: High

Description:

When fetching a directory using ipfs.get (or jsipfs get), it fails with the following error:

» jsipfs get QmXNHWdf9qr7A67FZQTFVb6Nr1Vfp4Ct3HXLgthGG61qy1
Saving file(s) QmXNHWdf9qr7A67FZQTFVb6Nr1Vfp4Ct3HXLgthGG61qy1
/Users/tmp/code/js-ipfs/src/cli/commands/files/get.js:67
      if (err) { throw err }
                 ^

Error: invalid node type
    at parseError (/Users/tmp/code/js-ipfs/node_modules/ipfs-api/src/utils/send-request.js:16:17)
    at ClientRequest.<anonymous> (/Users/tmp/code/js-ipfs/node_modules/ipfs-api/src/utils/send-request.js:39:14)
    at Object.onceWrapper (events.js:255:19)
    at ClientRequest.emit (events.js:160:13)
    at HTTPParser.parserOnIncomingClient (_http_client.js:548:21)
    at HTTPParser.parserOnHeadersComplete (_http_common.js:115:23)
    at Socket.socketOnData (_http_client.js:439:20)
    at Socket.emit (events.js:160:13)
    at addChunk (_stream_readable.js:269:12)
    at readableAddChunk (_stream_readable.js:256:11)

Steps to reproduce the error:

# start one daemon in one terminal
jsipfs daemon
# fetch in another terminal
jsipfs get QmXNHWdf9qr7A67FZQTFVb6Nr1Vfp4Ct3HXLgthGG61qy1

[daviddias]

go-ipfs has the same issue -- ipfs/kubo#4871 --. For some reason this went uncaught for a while.

[daviddias]

@achingbrain did you happen to make a fix for this when you implemented mfs and touched unixfs-engine?

[lidel]

Problem

I've just stumbled upon this bug while working on ipfs/ipfs-companion#533.

Tried running embedded js-ipfs node as the protocol handler backend, but it fails to load Turkish wikipedia mirror: /ipfs/QmT5NvUtoM5nWFfrQdVrFtvGfKFmG7AHE8P34isapyhCxX/wiki/Anasayfa.html where /wiki/ is hamt-sharded-directory.

What is interesting, our public gateways and local go-ipfs v4.0.17 load it without any problem, which implies support for resolving unixfs paths over HAMT sharded tree is possible.

How to Reproduce

Minimal repro of ipfs-companion use is basically this:

let cat = ipfs.files.catReadableStream('/ipfs/QmT5NvUtoM5nWFfrQdVrFtvGfKFmG7AHE8P34isapyhCxX/wiki/Anasayfa.html');
cat.on('data', (chunk) => {console.log(`Received ${chunk.length} bytes of data.`);});
cat.on('error', (err) => {console.log('catReadableStream produced error', err);});
cat.on('end', () => {console.log('There will be no more data.');});
cat.resume();

If you execute it, nothing will happen for a long time, but in the background js-ipfs-unixfs-engine:src/exporter/resolve.js becomes a hot path resolving /wiki/ over and over again until it hits a raw node such as /ipfs/QmT5NvUtoM5nWFfrQdVrFtvGfKFmG7AHE8P34isapyhCxX/I/favicon.png and src/exporter/object.js throws invalid node type due to pathRest being empty.

Not sure where to go from there, but I hope the above notes save some initial debug time.

[SgtPooki]

js-ipfs is being deprecated in favor of Helia. You can #4336 and read the migration guide.

Please feel to reopen with any comments by 2023-06-02. We will do a final pass on reopened issues afterwards (see #4336).

This issue is most likely resolved in Helia, please try it out!