Bring libp2p-websocket-star to the Transports family! 🌟 #122

mkg20001 · 2017-09-06T13:49:42Z

Adds tests for libp2p-websocket-star
Ref libp2p/js-libp2p-websocket-star#3

daviddias · 2017-09-08T07:05:46Z

circle.yml

    - sudo apt-get update
+    - sudo apt-get install -f || true
+    - sudo dpkg -i libnss3*.deb


@mkg20001 is this the fix for all the Chrome errors we started seeing in Circle?? Awesome!!!

//cc @victorbjelkholm

@diasdavid Yes.

daviddias · 2017-09-11T08:33:32Z

Ok, I've gone through and give a big CR (with refactoring in place) to websocket-star. Overall there is a ton of work there and I'm really happy to see it coming together, thank you @mkg20001 👏🏽👏🏽👏🏽👏🏽!

The CR+Refactor can be found in:

To release and integrate websocket-star with confidence, recommending our users to try it out, I still want to see some issues solved and/or improved. I've opened issues in both repos so that it is clear what is missing, lists can be found in:

@mkg20001 is this something you would like to finish? I can continue helping by providing CR and coding time, just not sure how much time I can allocate that soon :)

Thanks!

mkg20001 · 2017-09-11T12:20:40Z

@mkg20001 is this something you would like to finish? I can continue helping by providing CR and coding time, just not sure how much time I can allocate that soon :)

Yes but I'm also working on https://github.com/ZeroNetJS/zeronet-js/ and my last year in school just began so I won't be working on this project that much

daviddias · 2017-09-11T12:54:54Z

I didn't know about zeronet-js with libp2p! That's rad! :D

Actually, this is more than rad! it is super-duper rad! Wanna show it working today at the IPFS All Hands? ipfs/team-mgmt#499

mkg20001 · 2017-09-11T15:05:28Z

Actually, this is more than rad! it is super-duper rad! Wanna show it working today at the IPFS All Hands?

After thinking about that for a while and talking with my parents I decided it would be a good idea. I'm usually a bit shy and this would be the first every meeting/conference/etc where I would participate. But I would like to participate. How can I join? How does that work?

daviddias · 2017-10-23T13:53:41Z

@dryajov @victorbjelkholm can I get your review here as well?

victorb · 2017-10-23T15:08:18Z

package.json

@@ -60,6 +60,8 @@
    "libp2p-tcp": "~0.11.0",
    "libp2p-webrtc-star": "~0.13.0",
    "libp2p-websockets": "~0.10.1",
+    "libp2p-websocket-star": "0.3.0-rc4",
+    "libp2p-websocket-star-signal": "0.3.0-rc4",


Haven't seen that we are doing RC versions anywhere else, can we release stable versions of these?

mkg20001 · 2017-10-28T13:05:24Z

Rebased all changes against current master and released stable versions of the libp2p-websocket-star* modules.
Merge?

dryajov

LGTM

daviddias · 2017-10-29T15:44:28Z

.aegir.js

+    }
+    server2 = _server
+    ready()
+  })


@mkg20001 mind putting the two tasks, sigServer.start and wsRendezvous.start in a async.parallel flow?

They are already parallel...

I mean, to make it more structured with Parallel. I also see Ready being called 3 times and only checked for 2

daviddias · 2017-10-29T15:46:37Z

.aegir.js

-    }
-    server.stop(done)
-  }), 2000)
+  setTimeout(() => require('async/each')([node, server, server2], (s, n) => s.stop(n), done), 2000)


a) Please do not require modules throughout the codebase, always require it at the top.
b) Preserve the {} when nesting multiple calls.
c) Use async/parallel for this

daviddias · 2017-10-29T15:46:59Z

package.json

@@ -62,6 +62,8 @@
    "libp2p-tcp": "~0.11.1",
    "libp2p-webrtc-star": "~0.13.2",
    "libp2p-websockets": "~0.10.4",
+    "libp2p-websocket-star": "~0.5.0",
+    "libp2p-websocket-star-rendezvous": "~0.2.0",


Thank you for keeping the ~ for <1.0.0 :)

daviddias · 2017-10-29T15:48:22Z

test/browser-bundle/browser-bundle.js

@@ -36,11 +37,13 @@ class Node extends libp2p {
  constructor (peerInfo, peerBook, options) {
    options = options || {}
    const webRTCStar = new WebRTCStar()
+    const wsStar = new WebSocketStar({ id: peerInfo.id })


Does it still require that the PeerInfo gets passed on the constructor? Why?

in the README of ws-star it says: const ws = new WSStar({ id: id }) // the id is required for the crypto challenge

this requierment to have the { id: peerInfo.id } makes it impossible to configure the WebSocketStar through the IPFS node constructor like:

libp2p: { // add custom modules to the libp2p stack of your node modules: {} }

because the peer id is not available at that point

How else can the transport get the id?

The first question should be "why does this Id exist in the first place", if:

a) is to auth with the rendezvous point, the reality is that the rendezvous point doesn't have any list of pre-shared ids, so using the PeerId or something else at random is the same

b) is to auth with the other peers, then that is a double crypto challenge because secio already does that.

There are two things here:

a) Authentication between peers (end to end) - assured by SECIO ✔️

b) Authentication with the rendezvous point - what you are trying to achieve with the PeerId

b) is a bit more complicated because you want to peers to prove who they are with the rendezvous point.

I'm torn that one the best solution is through PeerId because that really breaks the transport interface expectation. We also want to move away from centralized rendezvous points and just make any node be able to do that, which kind of means that a transport will need access to libp2p itself. Once we end up doing this, it might lead to changing how modules are loaded to a DI scheme.

There are now 5 possible solutions:

Use wsstar.lazySetId() somewhere later in the code

Don't add the peerId and run the server with --disableCryptoChallenge (insecure!)

libp2p-websocket-star #117 (comment) (or similar)

DI scheme as mentioned above (isn't implemented so this isn't a good short-term solution)

Something else that works short-term at least

Which one would be the best? (I prefer the first one)

@mkg20001 what about deploying a first version without the cryptoChallenge so that js-ipfs users can try it out today and we work on the DI scheme (which @pgte and me will start using more of his time towards that) as a separate endeavor so that then we can add the crypto challenge?

@diasdavid The peerId IS accesible at the time of transport creation. No need to use the insecure version. https://github.com/ipfs/js-ipfs/compare/master...mkg20001:master.diff

@diasdavid Insecure server running at /dns/ws-star-signal-4.servep2p.com/wss/p2p-websocket-star/

daviddias · 2017-10-29T15:50:11Z

test/nodejs-bundle/tcp+websockets+websocket-star.js

+            expect(Object.keys(nodeWS.swarm.muxedConns)).to.have.length(1)
+            cb()
+          }
+        ], done)


All of the checks happening are synchronous, there is no reason to use async.parallel

I copied this from test/nodejs-bundle/tcp+websockets+webrtc-star.js ( https://github.com/libp2p/js-libp2p/blob/master/test/nodejs-bundle/tcp%2Bwebsockets%2Bwebrtc-star.js#L107-L118 )

Wanna improve both?

daviddias · 2017-10-29T15:50:21Z

test/nodejs-bundle/tcp+websockets+websocket-star.js

+            expect(Object.keys(peers)).to.have.length(1)
+            expect(Object.keys(nodeWS.swarm.muxedConns)).to.have.length(0)
+            cb()
+          }


daviddias · 2017-10-29T15:50:27Z

test/nodejs-bundle/tcp+websockets+websocket-star.js

+            expect(Object.keys(nodeAll.swarm.muxedConns)).to.have.length(1)
+            cb()
+          }
+        ], done)


daviddias · 2017-10-29T15:50:33Z

test/nodejs-bundle/tcp+websockets+websocket-star.js

+            expect(Object.keys(nodeWStar.swarm.muxedConns)).to.have.length(0)
+            cb()
+          }
+        ], done)


Beanow · 2017-11-16T18:52:11Z

Can you provide a configuration you've tested this with in js-ipfs?

In my tests to implement this I found some odd issues with the multi addresses.
Using a test server,

config: {
  Addresses: {
    Swarm: [
      '/dns4/ws-star-signal-2.servep2p.com/wss/p2p-websocket-star/'
    ]
  }
}

My logs output:

Swarm listening on /dns4/ws-star-signal-2.servep2p.com/wss/p2p-websocket-star/ipfs/QmcdoJPn4CStGjJd58XQE3GfLZQHSshMovhoGMNFoHd9px/ipfs/QmcdoJPn4CStGjJd58XQE3GfLZQHSshMovhoGMNFoHd9px

And it does not successfully connect with any peers over websocket star.

When I add this to src/core/runtime/libp2p-browser.js

peerInfo.multiaddrs.clear()
peerInfo.multiaddrs.add(multiaddr('/dns4/ws-star-signal-1.servep2p.com/wss/p2p-websocket-star/'))
const wsstar = new WSStar({id: peerInfo.id})

It will do peer discovery, bitswap and pubsub correctly.

mkg20001 · 2017-11-19T12:48:01Z

Try it with https://github.com/mkg20001/js-ipfs/tree/feat-websocketstar or apply this patch and see if it works (edit: works for me)

patrickdet · 2017-11-26T16:50:24Z

The only annoying thing is that now it is not possible to configure the libp2p transport with the WebSocketStar from the configuration like:

  const ipfs = new Ipfs({
    libp2p: {
      modules: {
        transport: [new WebSocketStar({id: peerInfo.id})],
        discovery: [new WebSocketStar({id: peerInfo.id})]
      }
    }
  })

because in that scenario the peerInfo is not available yet. Not sure if it is critical to have that though.

I'd really like to have the WebSocketStar available in Ipfs though, because the WebRTC keeps crashing my application.

mkg20001 · 2017-11-26T17:47:09Z

You can use it without peer id and use this signalling server instead: /dns/ws-star-signal-4.servep2p.com/wss/p2p-websocket-star/. note that it's not as secure as the other servers (no crypto challenge)

daviddias · 2017-11-26T22:19:31Z

Let's focus on figuring out #130 to get that PeerId on the transport. Until then let's move with the not as secure, also not yet audited version.

daviddias · 2017-11-26T22:19:56Z

.aegir.js

@@ -4,9 +4,12 @@ const Node = require('./test/nodejs-bundle/nodejs-bundle.js')
 const PeerInfo = require('peer-info')
 const PeerId = require('peer-id')
 const pull = require('pull-stream')
+const {parallel} = require('async')


Please use standard require('async/parallel')

daviddias · 2017-11-26T22:20:43Z

.aegir.js

 let server
+let server2


Since we now have two "servers", let's use qualifying names:

wrtcRendezvous

wsRendezvous

daviddias · 2017-11-26T22:20:56Z

.aegir.js

-  sigServer.start({ port: 15555 }, (err, _server) => {
+  sigServer.start({
+    port: 15555
+  }, (err, _server) => {


Unnecessary style change.

daviddias · 2017-11-26T22:22:44Z

.aegir.js

+    }
+    server2 = _server
+    ready()
+  })


I mean, to make it more structured with Parallel. I also see Ready being called 3 times and only checked for 2

daviddias · 2017-11-26T22:24:45Z

test/nodejs-bundle/tcp+websockets+webrtc-star.js

@@ -91,57 +91,32 @@ describe('TCP + WebSockets + WebRTCStar', () => {
      (cb) => nodeTCP.stop(cb),
      (cb) => nodeWS.stop(cb),
      (cb) => nodeWStar.stop(cb),
-      (cb) => ss.stop(done)
+      (cb) => ss.stop(cb)


good catch!

daviddias · 2017-11-26T22:26:28Z

test/nodejs-bundle/tcp+websockets+webrtc-star.js

-          }
-        ], done)
-      }
+      setTimeout(check, 500, nodeTCP, done, 1, 1)


Please use instead:

setTimeout(() => check(nodeTCP, 1, 1, done), 500

And update the function arguments respectively (rule: callback always comes last)

daviddias · 2017-11-26T22:28:02Z

test/nodejs-bundle/tcp+websockets+websocket-star.js

@@ -0,0 +1,154 @@
+/* eslint-env mocha */


Instead of creating a copy of tcp+websockets+webrtc-star please do try to augment the other test suite

daviddias · 2017-11-26T22:29:09Z

@mkg20001 thanks for pushing this :) I've made some comments. I am happy to finish this one tomorrow early morning if you don't have time until then. Thank you!

mitra42 · 2017-11-28T23:24:39Z

I see a number of configurations posted as not working above, is there a conclusion of a configuration that works ? Could it be posted here.

Co-authored-by: Chad Nehemiah <chad.nehemiah94@gmail.com>

## [1.1.6](libp2p/js-libp2p-webrtc@v1.1.5...v1.1.6) (2023-04-21) ### Bug Fixes * readme: Remove confusing section ([libp2p#122](libp2p/js-libp2p-webrtc#122)) ([dc78154](libp2p/js-libp2p-webrtc@dc78154))

mkg20001 requested a review from daviddias September 6, 2017 13:49

victorb added the status/in-progress In progress label Sep 6, 2017

mkg20001 mentioned this pull request Sep 6, 2017

Needs to be added to js-libp2p batch of tests libp2p/js-libp2p-websocket-star#3

Closed

mkg20001 force-pushed the libp2p-websocket-star branch from fe6c3a0 to 56675b9 Compare September 7, 2017 10:15

mkg20001 added status/ready Ready to be worked and removed status/in-progress In progress labels Sep 7, 2017

daviddias reviewed Sep 8, 2017

View reviewed changes

daviddias mentioned this pull request Sep 8, 2017

feat: add gateway route to daemon ipfs/js-ipfs#968

Merged

daviddias changed the title ~~test: add libp2p-websocket-star tests~~ Bring libp2p-websocket-star to the Transports family! 🌟 Sep 11, 2017

daviddias mentioned this pull request Sep 11, 2017

⚡️ v0.26.0 RELEASE 🚀 ipfs/js-ipfs#986

Closed

16 tasks

daviddias mentioned this pull request Sep 12, 2017

Firefox - crashes with excessive thread activity ipfs/js-ipfs#950

Closed

daviddias requested review from victorb and dryajov October 22, 2017 09:22

victorb reviewed Oct 23, 2017

View reviewed changes

mkg20001 added 2 commits October 28, 2017 14:55

test: add libp2p-websocket-star tests

614ec03

refactor: websocket-star-{signal => rendezvous}

0e2793f

mkg20001 force-pushed the libp2p-websocket-star branch from 56675b9 to 0e2793f Compare October 28, 2017 13:03

ghost assigned mkg20001 Oct 28, 2017

ghost added status/in-progress In progress and removed status/ready Ready to be worked labels Oct 28, 2017

dryajov approved these changes Oct 28, 2017

View reviewed changes

daviddias suggested changes Oct 29, 2017

View reviewed changes

misc: Apply requested changes

daf5b69

pgte mentioned this pull request Nov 8, 2017

dependency injection during boot time #130

Closed

Beanow mentioned this pull request Nov 18, 2017

Stable transports in the browser ipfs/js-ipfs#1088

Closed

daviddias mentioned this pull request Nov 26, 2017

⚡️ v0.27.0 RELEASE 🚀 ipfs/js-ipfs#1095

Closed

24 tasks

daviddias suggested changes Nov 26, 2017

View reviewed changes

daviddias added 2 commits November 27, 2017 07:36

chore: update deps

fe776a5

refactor: considerable amount of refactor, clean up test structure

47fdbdd

ghost assigned daviddias Nov 27, 2017

daviddias added 3 commits November 27, 2017 08:48

use right flag

baa560a

Merge branch 'master' into libp2p-websocket-star

0807954

test: refactor transport tests

724fe5b

daviddias approved these changes Nov 27, 2017

View reviewed changes

daviddias merged commit 95f029e into master Nov 27, 2017

daviddias deleted the libp2p-websocket-star branch November 27, 2017 09:10

ghost removed the status/in-progress In progress label Nov 27, 2017

This was referenced Nov 27, 2017

feat: WebSocketStar ipfs/js-ipfs#1090

Merged

Removing the Central Rendezvous Points for the *-star transports #134

Closed

maschad added a commit to maschad/js-libp2p that referenced this pull request Jun 21, 2023

fix: readme: Remove confusing section (libp2p#122)

dc78154

Co-authored-by: Chad Nehemiah <chad.nehemiah94@gmail.com>

Bring libp2p-websocket-star to the Transports family! 🌟 #122

Bring libp2p-websocket-star to the Transports family! 🌟 #122

Conversation

mkg20001 commented Sep 6, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

daviddias commented Sep 11, 2017

mkg20001 commented Sep 11, 2017

daviddias commented Sep 11, 2017 • edited Loading

mkg20001 commented Sep 11, 2017

daviddias commented Oct 23, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mkg20001 commented Oct 28, 2017

dryajov left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mkg20001 Nov 6, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Beanow commented Nov 16, 2017 • edited Loading

mkg20001 commented Nov 19, 2017 • edited Loading

patrickdet commented Nov 26, 2017 • edited Loading

mkg20001 commented Nov 26, 2017

daviddias commented Nov 26, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

daviddias commented Nov 26, 2017 • edited Loading

mitra42 commented Nov 28, 2017

daviddias commented Sep 11, 2017 •

edited

Loading

mkg20001 Nov 6, 2017 •

edited

Loading

Beanow commented Nov 16, 2017 •

edited

Loading

mkg20001 commented Nov 19, 2017 •

edited

Loading

patrickdet commented Nov 26, 2017 •

edited

Loading

daviddias commented Nov 26, 2017 •

edited

Loading