New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Awesome Endeavour: DHT Part II #856

Open
wants to merge 3 commits into
base: master
from

Conversation

Projects
None yet
5 participants
@daviddias
Member

daviddias commented May 19, 2017

This PR is the second part of the DHT integration in js-ipfs

Remaining tasks

Main work tracker: libp2p/js-libp2p-kad-dht#1

Needs the following PRs:

Blocked PRs:

@daviddias daviddias referenced this pull request Jun 13, 2017

Merged

Site refactor #52

32 of 33 tasks complete

@daviddias daviddias added ready and removed in progress labels Jul 3, 2017

@daviddias

This comment has been minimized.

Member

daviddias commented Jul 20, 2017

@daviddias

This comment has been minimized.

Member

daviddias commented Aug 25, 2017

It might actually be good to enable the DHT if it proves to not break things and then figure out the interop of the DHT with go-ipfs as we go.

@daviddias

This comment has been minimized.

Member

daviddias commented Aug 26, 2017

Updated deps including all the latest changes in Bitswap. DHT tests in JS land are fine. Still no love between JS and Go 😢

@daviddias

This comment has been minimized.

Member

daviddias commented Aug 26, 2017

CI is still not happy as well


  1) interface-ipfs-core tests .dht callback API .findpeer finds other peers:
     Uncaught AssertionError: expected [Error] to not exist
      at nodeA.dht.findpeer (node_modules/interface-ipfs-core/src/dht.js:90:32)
      at self._libp2pNode.peerRouting.findPeer (src/core/components/dht.js:81:18)
      at node_modules/async/internal/once.js:12:16
      at next (node_modules/async/waterfall.js:21:29)
      at node_modules/async/internal/onlyOnce.js:12:16
      at waterfall (node_modules/libp2p-kad-dht/src/index.js:449:20)
      at nextTask (node_modules/async/waterfall.js:16:14)
      at next (node_modules/async/waterfall.js:23:9)
      at node_modules/async/internal/onlyOnce.js:12:16
      at setImmediate (node_modules/multihashing-async/src/utils.js:8:7)
      at Immediate.<anonymous> (node_modules/async/internal/setImmediate.js:27:16)

  2) interface-ipfs-core tests .dht callback API .query returns the other node in the query:
     Uncaught AssertionError: expected [] to include 'QmS72d5n39h2zWXHQFFQs2Xub7GADbYimayvJYkocJv98D'
      at nodeA.dht.query (node_modules/interface-ipfs-core/src/dht.js:139:47)
      at self._libp2pNode._dht.getClosestPeers (src/core/components/dht.js:157:9)
      at q.run (node_modules/libp2p-kad-dht/src/index.js:172:18)
      at Query.run (node_modules/libp2p-kad-dht/src/query.js:47:14)
      at utils.convertBuffer (node_modules/libp2p-kad-dht/src/index.js:166:9)
      at setImmediate (node_modules/multihashing-async/src/utils.js:8:7)
      at Immediate.<anonymous> (node_modules/async/internal/setImmediate.js:27:16)

@ReisenB ReisenB referenced this pull request Sep 15, 2017

Open

State of IPFS #1

@daviddias daviddias added P0 - Critical ready and removed ready labels Oct 17, 2017

@daviddias daviddias referenced this pull request Nov 11, 2017

Open

Talking to js-ipfs #19

@RangerMauve RangerMauve referenced this pull request Jan 9, 2018

Open

pub/sub - publish / subscribe #64

0 of 3 tasks complete
@hoffmabc

This comment has been minimized.

hoffmabc commented Feb 16, 2018

Is this PR still up to date?

@daviddias

This comment has been minimized.

Member

daviddias commented Feb 16, 2018

@hoffmabc it is as with regards to status, DHT is still a WIP. Lot's of work happening on js-libp2p land itself -- https://github.com/libp2p/js-libp2p --. Currently focused on libp2p/js-libp2p#159

@dryajov dryajov referenced this pull request Mar 26, 2018

Merged

OKR 2018 Q2 #1281

@alanshaw

This comment has been minimized.

Member

alanshaw commented Apr 11, 2018

Is there any prior docs I can read about the technical challenges involved in go <-> js DHT interop?

@norzak norzak referenced this pull request Apr 15, 2018

Closed

WebTorrent DHT integration #1152

@daviddias daviddias referenced this pull request Apr 23, 2018

Merged

feat: ping #1299

2 of 2 tasks complete
@daviddias

This comment has been minimized.

Member

daviddias commented Apr 30, 2018

Let's unblock this endeavour with the delegated routing as soon as 0.29.0 gets released.

@vasco-santos

This comment has been minimized.

Member

vasco-santos commented Sep 18, 2018

I rebased this PR. With interface-ipfs-core#368 merged and released, I still have on test breaking. I will look at it later.

@vasco-santos

This comment has been minimized.

Member

vasco-santos commented Sep 18, 2018

@diasdavid

I found several problems that I will handle in the following days:

PRs for fixing this:
- js-libp2p-kad-dht#40
- js-libp2p#250

Regarding the first problem, I will create a PR shortly for fixing that.

Taking into account the third problem, I am not sure what is the way to fix that, as I think that we need to get data from both datastore and blockstore of the repo.

@daviddias

This comment has been minimized.

Member

daviddias commented Oct 26, 2018

@vasco-santos what's next for this PR?

@hoffmabc

This comment has been minimized.

hoffmabc commented Oct 26, 2018

Is dht interactivity with go nodes working?

@@ -44,7 +44,7 @@ module.exports = function libp2p (self) {
}
},
EXPERIMENTAL: {
dht: get(opts.options, 'EXPERIMENTAL.dht', false),
dht: true,

This comment has been minimized.

@daviddias

daviddias Oct 26, 2018

Member

It's fine to default to false to get this PR merged so that other folks can try it out

Show resolved Hide resolved README.md
@vasco-santos

This comment has been minimized.

Member

vasco-santos commented Oct 29, 2018

@diasdavid Regarding this PR, there is one problem to solve.

Taking into account interface-ipfs-core/js/src/dht/get.js#L48:

  • After creating a new Object, it will be propagated until the ipfs-bitswap, which will store the block and provide information to the network that it keeps the block js-ipfs-bitswap/index.js#L153.
  • libp2p-kad-dht receives the information but does not store the data, nor receives it at all js-libp2p-kad-dht/index.js#L409
  • According to the above, the block was stored in the repo's blockstore (ipfs-bitswap) and then, when a dht/get operation happens, the other node will ask the previous node about the data that he claimed before to have, but it will not have it on its datastore js-libp2p-kad-dht/private.js#L347
  • The datastore provided by js-libp2p is not available js-libp2p/index.js#L92 and kad-dht uses a MemoryStore.

@diasdavid What should be the expected behavior here?

I think that once this issue is solved, we can move this PR forward, since it is below the experimental flag and people can start testing it.

Meanwhile, I will be working in the next days on some stress tests to the DHT, in order to identify possible issues.



@diasdavid @hoffmabc

After interop#36 gets merged, DHT should be interoperable with go-ipfs. This PR is ready for being reviewed.

@vasco-santos vasco-santos referenced this pull request Oct 29, 2018

Open

fix: update dht tests #383

0 of 1 task complete
@vasco-santos

This comment has been minimized.

Member

vasco-santos commented Oct 29, 2018

@diasdavid in the context of my above comment, I created a WIP PR interface-ipfs-core/pull/383 skipping that test and adding a new test for get and for put. Let me know, what should be the expected from that problem for me to solve it.

@daviddias

This comment has been minimized.

Member

daviddias commented Oct 31, 2018

image

@vasco-santos

This comment has been minimized.

Member

vasco-santos commented Oct 31, 2018

I completelly agree and just created a PR js-libp2p/pull/270 for adding the datastore through the config.

According to your remaining comments, I understand that this test interface-ipfs-core/dht/get.js#L48-L64 is not a valid test for the dht, as the object is only stored in the Blockstore. Can you confirm that my understanding is correct @diasdavid ?

I have a PR interface-ipfs-core#383/files#diff-fb6bf8fdc8bd42ae590a94dab7a95bafR66 where I added a new dht.get test, and I think that I can remove the other one that I mentioned above.

@daviddias

This comment has been minimized.

Member

daviddias commented Oct 31, 2018

Can you confirm that my understanding is correct @diasdavid ?

It is not.

When you add a object, a record must be added on the provide step. In that test, a unixfs-dir object is created, added to IPFS and then another node does a dht.get to get that provider record.

@vasco-santos

This comment has been minimized.

Member

vasco-santos commented Nov 2, 2018

Can you confirm that my understanding is correct @diasdavid ?

It is not.

When you add a object, a record must be added on the provide step. In that test, a unixfs-dir object is created, added to IPFS and then another node does a dht.get to get that provider record.

First, I thought about adding the data into the provide js-ipfs-bitswap/network.js#L119. This way (libp2p.contentRouting.provide(cid, data, callback) ), if provide received a value, it should be added to the dht. The test was ok in this way, but this felt wrong to me, as we were already storing the data in the blockstore.

After exploring ipfs-bitswap, more precisely js-ipfs-bitswap/test/bitswap.js#L137-L148 and js-ipfs-bitswap/src/index.js#L223, I figured out that the test interface-ipfs-core/js/src/dht/get.js#L48 should be something as follows instead:

waterfall([
 (cb) => nodeB.object.new('unixfs-dir', cb),
 (dagNode, cb) => setTimeout(() => cb(null, dagNode), 20000),
 (dagNode, cb) => {
   const multihash = dagNode.toJSON().multihash

   // nodeA.dht.get(multihash, cb)
   nodeA.object.get(multihash, cb)
  },
](

This way, the node discovery should be done through the DHT and after that they are able to exchange their blocks across the network.

klueq added a commit to klueq/klueq.github.io that referenced this pull request Nov 3, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment