Awesome Endeavour: DHT Part II

[daviddias]

This PR is the second part of the DHT integration in js-ipfs

Remaining tasks

Main work tracker: libp2p/js-libp2p-kad-dht#1

• Make usage of the DHT efficient (avoid doing multiple query calls for the same file)
  • add .getMany js-ipfs-bitswap - ipfs/js-ipfs-bitswap#163
  • add .getMany js-ipfs-block-service - ipfs/js-ipfs-block-service#79 ipfs/js-ipfs-block-service#81
  • add .get() to js-ipld ipld/js-ipld#132
  • add .get() from the DAG API to js-ipfs-unixfs-engine - ipfs/js-ipfs-unixfs-engine#216
• interface-ipfs-core
  • implement remaining DHT methods in js-ipfs-api
  • Update tests and spec, ensure that both pass in js-ipfs and js-ipfs-api
  • Augment the test suite and spec
    • first pass ipfs/interface-ipfs-core#288
  • Improve the API ipfs/interface-ipfs-core#136 (non critical)
• js-libp2p
  • Make configuring a Node nicer and support multiple Peer Routing + Content Routing systems libp2p/js-libp2p#166
  • libp2p.next() libp2p/js-libp2p#159 (parts needed are shipped)
  • js-libp2p-kad-dht
    • Implement it
    • Integrate in js-libp2p js-libp2p libp2p/js-libp2p#86
    • Integrate in js-libp2p-ipfs-nodejs https://github.com/ipfs/js-libp2p-ipfs-nodejs
    • Integrate libp2p-ipfs-browser will only add DHT there after Relay is in
    • Make random walk be a discovery service libp2p/js-libp2p-kad-dht#7
  • Delegated Routing libp2p/js-libp2p#120
• Implement .dht in js-ipfs
  • core api
  • http-api
  • cli
• interop tests
  • js node in go-ipfs-land (implemented, not green)
  • go node in js-ipfs-land
• integration tests ipfs/notes#294

Needs the following PRs:

• js-libp2p@0.24.0
• ipfs/interface-ipfs-core#389
• ipfs/js-ipfs-api#890

Blocked PRs:

• ipfs/interface-ipfs-core#383
• ipfs/interop#36

[daviddias]

• ipfs/js-ipfs-bitswap#142

[daviddias]

It might actually be good to enable the DHT if it proves to not break things and then figure out the interop of the DHT with go-ipfs as we go.

[daviddias]

Updated deps including all the latest changes in Bitswap. DHT tests in JS land are fine. Still no love between JS and Go 😢

[daviddias]

CI is still not happy as well

  1) interface-ipfs-core tests .dht callback API .findpeer finds other peers:
     Uncaught AssertionError: expected [Error] to not exist
      at nodeA.dht.findpeer (node_modules/interface-ipfs-core/src/dht.js:90:32)
      at self._libp2pNode.peerRouting.findPeer (src/core/components/dht.js:81:18)
      at node_modules/async/internal/once.js:12:16
      at next (node_modules/async/waterfall.js:21:29)
      at node_modules/async/internal/onlyOnce.js:12:16
      at waterfall (node_modules/libp2p-kad-dht/src/index.js:449:20)
      at nextTask (node_modules/async/waterfall.js:16:14)
      at next (node_modules/async/waterfall.js:23:9)
      at node_modules/async/internal/onlyOnce.js:12:16
      at setImmediate (node_modules/multihashing-async/src/utils.js:8:7)
      at Immediate.<anonymous> (node_modules/async/internal/setImmediate.js:27:16)

  2) interface-ipfs-core tests .dht callback API .query returns the other node in the query:
     Uncaught AssertionError: expected [] to include 'QmS72d5n39h2zWXHQFFQs2Xub7GADbYimayvJYkocJv98D'
      at nodeA.dht.query (node_modules/interface-ipfs-core/src/dht.js:139:47)
      at self._libp2pNode._dht.getClosestPeers (src/core/components/dht.js:157:9)
      at q.run (node_modules/libp2p-kad-dht/src/index.js:172:18)
      at Query.run (node_modules/libp2p-kad-dht/src/query.js:47:14)
      at utils.convertBuffer (node_modules/libp2p-kad-dht/src/index.js:166:9)
      at setImmediate (node_modules/multihashing-async/src/utils.js:8:7)
      at Immediate.<anonymous> (node_modules/async/internal/setImmediate.js:27:16)

[hoffmabc]

Is this PR still up to date?

[daviddias]

@hoffmabc it is as with regards to status, DHT is still a WIP. Lot's of work happening on js-libp2p land itself -- https://github.com/libp2p/js-libp2p --. Currently focused on libp2p/js-libp2p#159

[alanshaw]

Is there any prior docs I can read about the technical challenges involved in go <-> js DHT interop?

[daviddias]

Let's unblock this endeavour with the delegated routing as soon as 0.29.0 gets released.

[vasco-santos]

I rebased this PR. With interface-ipfs-core#368 merged and released, I still have on test breaking. I will look at it later.

[vasco-santos]

@diasdavid

I found several problems that I will handle in the following days:

• js-libp2p
  • libp2p/js-libp2p/dht.js#L12 should have the timeout parameter
    • js-ipfs sends it
    • js-libp2p-kad-dhtindex.js#L186 receives it
    • This will result in loosing the callback on the way.

PRs for fixing this:
- js-libp2p-kad-dht#40
- js-libp2p#250

• js-libp2p-kad-dht/index.js#L118 needs to have randomWalk started.

  • I did not see it being started anywhere else.

• interface-ipfs-core get test failing interface-ipfs-core/dht/get.js#L48-L64

  • After creating a new Object, it will be propagated until the ipfs-bitswap, which will store the block and provide information to the network that it keeps the block js-ipfs-bitswap/index.js#L153.
  • libp2p-kad-dht receives the information but does not store the data, nor receives it at all js-libp2p-kad-dht/index.js#L409
  • According to the above, the block was stored in the repo's blockstore (ipfs-bitswap) and then, when a dht/get operation happens, the other node will ask the previous node about the data that he claimed before to have, but it will not have it on its datastore js-libp2p-kad-dht/private.js#L347
  • The datastore provided by js-libp2p is not available js-libp2p/index.js#L92 and kad-dht uses a MemoryStore.

Regarding the first problem, I will create a PR shortly for fixing that.

Taking into account the third problem, I am not sure what is the way to fix that, as I think that we need to get data from both datastore and blockstore of the repo.

[daviddias]

@vasco-santos what's next for this PR?

[hoffmabc]

Is dht interactivity with go nodes working?

[daviddias]

It's fine to default to false to get this PR merged so that other folks can try it out

[vasco-santos]

@diasdavid Regarding this PR, there is one problem to solve.

Taking into account interface-ipfs-core/js/src/dht/get.js#L48:

• After creating a new Object, it will be propagated until the ipfs-bitswap, which will store the block and provide information to the network that it keeps the block js-ipfs-bitswap/index.js#L153.
• libp2p-kad-dht receives the information but does not store the data, nor receives it at all js-libp2p-kad-dht/index.js#L409
• According to the above, the block was stored in the repo's blockstore (ipfs-bitswap) and then, when a dht/get operation happens, the other node will ask the previous node about the data that he claimed before to have, but it will not have it on its datastore js-libp2p-kad-dht/private.js#L347
• The datastore provided by js-libp2p is not available js-libp2p/index.js#L92 and kad-dht uses a MemoryStore.

@diasdavid What should be the expected behavior here?

I think that once this issue is solved, we can move this PR forward, since it is below the experimental flag and people can start testing it.

Meanwhile, I will be working in the next days on some stress tests to the DHT, in order to identify possible issues.

---

---

@diasdavid @hoffmabc

After interop#36 gets merged, DHT should be interoperable with go-ipfs. This PR is ready for being reviewed.

[vasco-santos]

@diasdavid in the context of my above comment, I created a WIP PR interface-ipfs-core/pull/383 skipping that test and adding a new test for get and for put. Let me know, what should be the expected from that problem for me to solve it.

[daviddias]

[vasco-santos]

I completelly agree and just created a PR js-libp2p/pull/270 for adding the datastore through the config.

According to your remaining comments, I understand that this test interface-ipfs-core/dht/get.js#L48-L64 is not a valid test for the dht, as the object is only stored in the Blockstore. Can you confirm that my understanding is correct @diasdavid ?

I have a PR interface-ipfs-core#383/files#diff-fb6bf8fdc8bd42ae590a94dab7a95bafR66 where I added a new dht.get test, and I think that I can remove the other one that I mentioned above.

[daviddias]

Can you confirm that my understanding is correct @diasdavid ?

It is not.

When you add a object, a record must be added on the provide step. In that test, a unixfs-dir object is created, added to IPFS and then another node does a dht.get to get that provider record.

[vasco-santos]

Can you confirm that my understanding is correct @diasdavid ?

It is not.

When you add a object, a record must be added on the provide step. In that test, a unixfs-dir object is created, added to IPFS and then another node does a dht.get to get that provider record.

First, I thought about adding the data into the provide js-ipfs-bitswap/network.js#L119. This way (libp2p.contentRouting.provide(cid, data, callback) ), if provide received a value, it should be added to the dht. The test was ok in this way, but this felt wrong to me, as we were already storing the data in the blockstore.

After exploring ipfs-bitswap, more precisely js-ipfs-bitswap/test/bitswap.js#L137-L148 and js-ipfs-bitswap/src/index.js#L223, I figured out that the test interface-ipfs-core/js/src/dht/get.js#L48 should be something as follows instead:

waterfall([
 (cb) => nodeB.object.new('unixfs-dir', cb),
 (dagNode, cb) => setTimeout(() => cb(null, dagNode), 20000),
 (dagNode, cb) => {
   const multihash = dagNode.toJSON().multihash

   // nodeA.dht.get(multihash, cb)
   nodeA.object.get(multihash, cb)
  },
](

This way, the node discovery should be done through the DHT and after that they are able to exchange their blocks across the network.