Awesome Endeavour: DHT Part II #856

daviddias · 2017-05-19T14:54:58Z

daviddias · 2017-07-20T05:45:19Z

ipfs/js-ipfs-bitswap#142

daviddias · 2017-08-25T15:25:03Z

It might actually be good to enable the DHT if it proves to not break things and then figure out the interop of the DHT with go-ipfs as we go.

daviddias · 2017-08-26T14:31:57Z

Updated deps including all the latest changes in Bitswap. DHT tests in JS land are fine. Still no love between JS and Go 😢

daviddias · 2017-08-26T14:32:07Z

CI is still not happy as well


  1) interface-ipfs-core tests .dht callback API .findpeer finds other peers:
     Uncaught AssertionError: expected [Error] to not exist
      at nodeA.dht.findpeer (node_modules/interface-ipfs-core/src/dht.js:90:32)
      at self._libp2pNode.peerRouting.findPeer (src/core/components/dht.js:81:18)
      at node_modules/async/internal/once.js:12:16
      at next (node_modules/async/waterfall.js:21:29)
      at node_modules/async/internal/onlyOnce.js:12:16
      at waterfall (node_modules/libp2p-kad-dht/src/index.js:449:20)
      at nextTask (node_modules/async/waterfall.js:16:14)
      at next (node_modules/async/waterfall.js:23:9)
      at node_modules/async/internal/onlyOnce.js:12:16
      at setImmediate (node_modules/multihashing-async/src/utils.js:8:7)
      at Immediate.<anonymous> (node_modules/async/internal/setImmediate.js:27:16)

  2) interface-ipfs-core tests .dht callback API .query returns the other node in the query:
     Uncaught AssertionError: expected [] to include 'QmS72d5n39h2zWXHQFFQs2Xub7GADbYimayvJYkocJv98D'
      at nodeA.dht.query (node_modules/interface-ipfs-core/src/dht.js:139:47)
      at self._libp2pNode._dht.getClosestPeers (src/core/components/dht.js:157:9)
      at q.run (node_modules/libp2p-kad-dht/src/index.js:172:18)
      at Query.run (node_modules/libp2p-kad-dht/src/query.js:47:14)
      at utils.convertBuffer (node_modules/libp2p-kad-dht/src/index.js:166:9)
      at setImmediate (node_modules/multihashing-async/src/utils.js:8:7)
      at Immediate.<anonymous> (node_modules/async/internal/setImmediate.js:27:16)

hoffmabc · 2018-02-16T15:28:19Z

Is this PR still up to date?

daviddias · 2018-02-16T17:11:03Z

@hoffmabc it is as with regards to status, DHT is still a WIP. Lot's of work happening on js-libp2p land itself -- https://github.com/libp2p/js-libp2p --. Currently focused on libp2p/js-libp2p#159

alanshaw · 2018-04-11T11:39:31Z

Is there any prior docs I can read about the technical challenges involved in go <-> js DHT interop?

daviddias · 2018-04-30T19:18:41Z

Let's unblock this endeavour with the delegated routing as soon as 0.29.0 gets released.

alanshaw

Additionally the CLI command handlers may soon need to be updated to use getIpfs instead of ipfs. See #1860 for details.

alanshaw · 2019-02-05T16:44:45Z

README.md

@@ -319,7 +314,6 @@ Enable and configure experimental features.
 - `pubsub` (boolean): Enable libp2p pub-sub. (Default: `false`)
 - `ipnsPubsub` (boolean): Enable pub-sub on IPNS. (Default: `false`)
 - `sharding` (boolean): Enable directory sharding. Directories that have many child objects will be represented by multiple DAG nodes instead of just one. It can improve lookup performance when a directory has several thousand files or more. (Default: `false`)
- `dht` (boolean): Enable KadDHT. **This is currently not interoperable with `go-ipfs`.**


OK, so, I'm half inclined to just enable it in the browser by default, expose ourselves to the issues and get them fixed asap. That said, since this is such a big feature perhaps not having it enabled by default in the browser initially will give us some scope to test it out first and fix the bigger issues, so that we don't expose all our users to the bugs and make everyone mad.

We do need a way to enable/disable it though - if the dht config option is being removed from EXPERIMENTAL, is it being added to the top level?

alanshaw · 2019-02-05T16:44:45Z

src/cli/commands/dht/find-peer.js

+      const peers = await ipfs.dht.findPeer(peerID)
+      const addresses = peers.multiaddrs.toArray().map((ma) => ma.toString())
+
+      print(addresses)


print is just console.log which will give you output like:

[ '/ip4/127.0.0.1/tcp/4001', '/ip4/169.254.83.88/tcp/4001', '/ip4/192.168.1.95/tcp/4001' ]

In comparison go-ipfs outputs:

$ ipfs dht findpeer QmZMxNdpMkewiVZLMRxaNxUeZpDUb34pWjZ1kZvsd16Zic /ip6/2001:19f0:5:66d1::/tcp/4001 /ip4/149.28.228.162/tcp/33154 /ip4/45.77.150.82/tcp/4001 /ip6/::1/tcp/4001 /ip6/2001:19f0:5:5d40:5400:1ff:fec1:ce94/tcp/4001 /ip4/127.0.0.1/tcp/4001 /ip4/149.28.228.162/tcp/4001 /ip4/127.0.0.1/tcp/8081/ws

alanshaw · 2019-02-05T16:44:45Z

src/cli/commands/dht/provide.js

+  },
+
+  handler ({ ipfs, key, resolve }) {
+    // TODO add recursive option


Please pass this option to ipfs.dht.provide so it can throw and inform the user the option is not implemented yet.

alanshaw · 2019-02-05T16:44:45Z

src/core/components/libp2p.js

@@ -76,6 +76,9 @@ function defaultBundle ({ datastore, peerInfo, peerBook, options, config }) {
        }
      },
      dht: {
+        kBucketSize: get(options, 'dht.kBucketSize', 20),
+        enabled: get(options, 'dht.enabled', true) && !(get(options, 'local', false)),


"local" is now "offline"?

alanshaw · 2019-02-05T16:44:45Z

src/core/components/pin-set.js

-const some = require('async/some')
-const eachOf = require('async/eachOf')
+const someSeries = require('async/someSeries')
+const eachOfSeries = require('async/eachOfSeries')


What does this change do?

Using some is making a lot of parallel dht find and connect requests.

So, everything will get queried even if we find the target node, as the find takes longer than initiating the requests.

In the long run, we should refactor the pin-set, in order to try to parallelize some operations, but this is the safest option currently.

alanshaw · 2019-02-05T16:44:45Z

src/http/api/resources/dht.js

+      arg: Joi.string().required()
+    }).unknown()
+  },
+  handler: (request, reply) => {


These handlers now need to change for Hapi 18 - please shout if you have any questions!

daviddias added awesome endeavour status/in-progress labels May 19, 2017

daviddias mentioned this pull request Jun 13, 2017

Site refactor libp2p/website#52

Merged

32 of 33 tasks complete

daviddias added status/ready and removed status/in-progress labels Jul 3, 2017

This was referenced Jul 5, 2017

Uploaded file via Spawned Node not accessible via public gateway #886

Closed

Implement IPNS #209

Closed

daviddias mentioned this pull request Aug 25, 2017

The file added by node.js can't be get/cat on other nodes #957

Closed

daviddias force-pushed the feat/dht-part-ii branch from 6a29c16 to df424e5 Aug 26, 2017

ReisenB mentioned this pull request Sep 15, 2017

State of IPFS DistributedMemetics/DM#1

Open

daviddias added P0 status/ready and removed status/ready labels Oct 17, 2017

daviddias mentioned this pull request Nov 11, 2017

Talking to js-ipfs richardschneider/net-ipfs-http-client#19

Open

RangerMauve mentioned this pull request Jan 9, 2018

pub/sub - publish / subscribe ipfs/notes#64

Open

0 of 3 tasks complete

dryajov mentioned this pull request Mar 26, 2018

OKR 2018 Q2 #1281

Merged

norzak mentioned this pull request Apr 15, 2018

WebTorrent DHT integration webtorrent/webtorrent#1152

Open

daviddias mentioned this pull request Apr 23, 2018

feat: ping #1299

Merged

2 of 2 tasks complete

fazo96 mentioned this pull request May 2, 2018

peers don't connect automatically through circuit, but they do through websocket-star rendezvous server #1309

Closed

vasco-santos mentioned this pull request May 8, 2018

Merge branch 'master' of linonetwo/ipfs-browser-gateway ipfs-shipyard/service-worker-gateway#1

Closed

daviddias self-assigned this May 30, 2018

daviddias force-pushed the feat/dht-part-ii branch from df424e5 to 5347ffd May 30, 2018

alanshaw requested changes Feb 5, 2019

View changes

vasco-santos force-pushed the feat/dht-part-ii branch 2 times, most recently from 999ebf2 to 2b548be Feb 6, 2019

alanshaw mentioned this pull request Feb 6, 2019

dht functionality #1857

Closed

vasco-santos force-pushed the feat/dht-part-ii branch 2 times, most recently from 9a4083e to c16bdc4 Feb 6, 2019

alanshaw and others added 9 commits Dec 11, 2018

chore: increase silent timeout even more

53a3f32

feat: dht ready

28f5641

test: enable dht core test

8ecb7b5

feat: add dht cli

f9cd739

chore: enable dht by default

eadd7d5

fix: code review

4453b55

fix: code review

2a76f98

fix: ci tests

045e305

fix: code review

Loading status checks…

2956637

vasco-santos force-pushed the feat/dht-part-ii branch from c16bdc4 to 2956637 Feb 6, 2019

alanshaw approved these changes Feb 6, 2019

View changes

Merge branch 'master' into feat/dht-part-ii

Verified

This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.

GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits

Loading status checks…

2651787

ghost assigned alanshaw Feb 7, 2019

vasco-santos force-pushed the feat/dht-part-ii branch 4 times, most recently from 0c73e99 to 147b100 Feb 7, 2019

fix: make ci happy again

Loading status checks…

02d1704

vasco-santos force-pushed the feat/dht-part-ii branch from 147b100 to 02d1704 Feb 7, 2019

vasco-santos mentioned this pull request Feb 8, 2019

feat: limit connections number #1872

Merged

4 of 4 tasks complete

alanshaw deleted the feat/dht-part-ii branch Feb 8, 2019

ghost removed the status/in-progress label Feb 8, 2019

ipfs / js-ipfs

Awesome Endeavour: DHT Part II #856

Awesome Endeavour: DHT Part II #856

daviddias commented May 19, 2017 •

edited by vasco-santos

daviddias commented Jul 20, 2017 •

edited

daviddias commented Aug 25, 2017

daviddias commented Aug 26, 2017

daviddias commented Aug 26, 2017

hoffmabc commented Feb 16, 2018

daviddias commented Feb 16, 2018

alanshaw commented Apr 11, 2018

daviddias commented Apr 30, 2018

alanshaw left a comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

ipfs / js-ipfs

Awesome Endeavour: DHT Part II #856

Awesome Endeavour: DHT Part II #856

Conversation

daviddias commented May 19, 2017 • edited by vasco-santos

Remaining tasks

Needs the following PRs:

Blocked PRs:

daviddias commented Jul 20, 2017 • edited

daviddias commented Aug 25, 2017

daviddias commented Aug 26, 2017

daviddias commented Aug 26, 2017

hoffmabc commented Feb 16, 2018

daviddias commented Feb 16, 2018

alanshaw commented Apr 11, 2018

daviddias commented Apr 30, 2018

alanshaw left a comment

This comment has been minimized.

alanshaw Feb 5, 2019 Member

This comment has been minimized.

alanshaw Feb 5, 2019 Member

This comment has been minimized.

alanshaw Feb 5, 2019 Member

This comment has been minimized.

alanshaw Feb 5, 2019 Member

This comment has been minimized.

alanshaw Feb 5, 2019 Member

This comment has been minimized.

vasco-santos Feb 6, 2019 Member

This comment has been minimized.

alanshaw Feb 5, 2019 Member

daviddias commented May 19, 2017 •

edited by vasco-santos

daviddias commented Jul 20, 2017 •

edited

alanshaw Feb 5, 2019
Member

alanshaw Feb 5, 2019
Member

alanshaw Feb 5, 2019
Member

alanshaw Feb 5, 2019
Member

alanshaw Feb 5, 2019
Member

vasco-santos Feb 6, 2019
Member

alanshaw Feb 5, 2019
Member