WebRTC DHT

[piedshag]

Discussion for any developments with the webrtc dht

[astro]

https://www.bountysource.com/issues/9787231-dht?utm_campaign=plugin&utm_content=tracker%2F298985&utm_medium=issues&utm_source=github

[feross]

This is something to look at for anyone looking to implement this: https://software.intel.com/en-us/blogs/2015/03/18/meshcentral-experimental-webrtc-mesh

cc @jhiesey

[lmatteis]

perhaps this is related https://github.com/diasdavid/webrtc-explorer

[smarts]

I was looking for a [Google] Chrome Browser extension for handling BitTorrent metainfo files and magnet URNs and I stumbled across this project, which seems really cool! If I understand correctly DHT is required for the handling of magnet links that don't include tracker URLs, correct? If so does that mean webtorrent doesn't work in the browser for magnet URNs w/o webrtc-dht?

[feross]

@smarts When we detect a magnet link without a tracker on https://instant.io, we just add the tracker wss://tracker.webtorrent.io so that there's a place to get peers from. You can do the same in your code. :-)

[feross]

https://github.com/hallettj/bitable

[smarts]

Thanks @feross!

[fsteff]

Any progress on this? I really need this feature (in combination with mutable keys, but one after the other...)
I´ll take a look at what @fbodz has already done and try to support him.

[feross]

No progress. If someone wants to work on this, the WebTorrent project would greatly benefit!

[iaaaan]

Same here, I'm very excited about this. I added $35 to the bounty... it's very few but I hope it might help in some way.

[moshest]

I'm doing research about it a couple of hours and I think that DHT algorithm isn't fit right for WebRTC:

1. Each connection channel is only one way so each DHT request will cost us two channels.
2. Each channel should remain open for future requests, otherwise we will need to build the n-buckets database all over again.
3. Each connection require offer-answer handshake via singling server or third peer.

It's seems that WebRTC is very expensive for DHT requests.
Any suggestions for maybe other algorithm?

[jhiesey]

I actually spent some time thinking about this problem a few weeks ago and I think a WebRTC DHT may be feasible. I'll add my thoughts here soon.

[daviddias]

@feross what would be necessary requisites to have a DHT interesting enough to be used for the WebTorrent project. I'm asking this with the following thoughts in mind:

• Implementing Kademlia DHT for browsers would be quite expensive, even unfeasible if we keep open connections to the older peers (browsers would go quickly out of resources)
• BitTorrent performs DHT searches using a iterative method (we ask peer A, and then peer B), this would mean that adding to the WebRTC DataChannels opened with our KBucket, we would be also be signalling on the fly more DataChannels to the peers we learn along the way to find the destination. Another way to perform the search is "Store and forward", which is how webrtc-explorer works (also IP, or DNS in recursive mode).
• Do we have to use Kademlia at all? After all if we keep the same ID namespace and deterministic properties of a Structured DHT, we could have on the browser side a different algorithm that is more considered of browser capabilities/limitations.
• Is it expected for a bittorrent client to be able to stream a file being served by a browser? Well, only with the hybrid client in the middle to do the "providing" step, correct?

Happy to work of these things for the WebTorrent project :) Please free to push any notes, ideas or any other info you have thought about this.

[moshest]

After some more thinking, I think that the signaling server is the best place for doing DHT queries. Webtorrent should have a few open signaling servers exactly for that case.

Please take a look at PeerJS. We can change the server implementation and add support for DHT queries between those servers.

[feross]

@diasdavid

I was thinking we could adapt the Kademila algorithm. So, normally the bucket size is K=8 or K=16. But we only need that level of redundancy in each bucket because there are no guarantees when you add a DHT node to the routing table that they'll be online when you need to contact them later. With WebRTC, we have to keep the connection active. This means we can reduce the number of nodes in each bucket to something lower, like K=2.

In traditional Kademlia (K=8), for an network of 1M nodes (evenly distributed across the id space), each node will have 144 entries in routing table (check my math).

For K=2 and a network of 1M nodes, we only need 40 entries in the routing table, a much more reasonable number of data channels. :)

Another difference caused by WebRTC is that when other people have you in their routing table, you have to keep an open connection to them so they can contact you. These are "wasted" connections for you, since they aren't necessary for your own routing table. Nonetheless, they need to be kept open for the other node's sake.

If a node is equally likely to be in every other node's routing table, then that would add 40 additional connections. But, over time Kademlia actually prefers long-running nodes since the client never evicts a responsive node from a bucket. This means that over time long-running nodes will end up in lots of client's routing tables. With the WebRTC model, that means that the long-running nodes will end up needing to keep tons of connections open to support other people's routing tables.

To solve this, I think we could change the part of Kademlia where we keep longer-running nodes in the table, so every node becomes equally likely to be added to the routing table.

Of course, we could ditch Kademlia entirely and come up with something better from scratch, but then it will be harder for existing torrent clients to add support. So something that's basically Kademlia with a few changes would be easier for them to support.

For WebTorrent, it doesn't matter. We can add anything that works. If there was a reliable general purpose WebRTC DHT not based on Kademlia, WebTorrent would start using it immediately. Even if desktop torrent clients don't support it, it's still useful for web peers to find each other. And later, if a WebRTC DHT that is closer to Kademlia came along, we could always switch to it later :)

"Store and forward" might be a good change for WebRTC since it eliminates the STUN/ICE connection overhead that would happen for each connection.

It'd be awesome to hear your thoughts on these ideas, @diasdavid!

[feross]

@moshest Putting that logic into a central server would mean that it's no longer distributed, the D in DHT. At that point, it's basically a tracker server, and we already have that part working :) What we really want is a DHT so we can eliminate the central point of failure that is a tracker server.

[daviddias]

Thanks @feross , lots of insights! That was exactly what I was looking for, to kick things off :)

I think a K bucket of 2 is a good bet on a WebRTC DHT, or I would even say that a K bucket of 1 should suffice. If I'm not missing anything, K buckets offer mainly redundancy for greater availability, having more than one peer at a specific distance means that we always have more than one shot to contact someone on that branch/bucket, which is great for non browser DHTs, since peer representations are pairs IP:Port. In a browser DHT, since we have to keep Data Channels open to avoid all of the STUN/ICE dance each time, we know which is the precise moment a peer goes away, giving us the chance to react and quickly do the handshake with a new peer.

Since we need to limit the number of "xor-metric distances" (going with "xmd" for short) used, so that we avoid having too many DataChannels open, we can consider some strategies for finger distribution, for e.g:

• Inspired on CHORD/PASTRY, giving privilege to closer distances and ensuring that at least we have some fingers to make long jumps.
• Uniform gap, simpler, for e.g with a cap of 20 xmd and for a DHT with 160 bit, we would have entries for 8 bit xmd, 16 bit xmd, 24 bit xmd, 32, 40, 48 ... 160 and one kbucket for each..

The first case is interesting for things like PAST (or DynamoDB), because peers store data ( and not pointers), so enables data replication to closer peers. But since this is WebTorrent and we only need to do is know who is 'providing' what, the second might make more sense. Dunno, ideas :D

One thing to have in mind though is that by having a xmd cap of 20 and K=2, we only set the minimum data channels needed, there still might be some cases where a peer is far enough to not be picked by another peer k bucket , but since it has to know some peer on that branch, it will open a connection to the peer that didn't picked him, creating kind of a one direction channel.

For WebTorrent, it doesn't matter. We can add anything that works. If there was a reliable general purpose WebRTC DHT not based on Kademlia, WebTorrent would start using it immediately

Independently of which the DHT is based on, there will always have to be something to do the Signalling. but for WebTorrent case, you already have that. Would it be interesting that instead of starting two signalling servers, we could have a way to mux between signalling for peers that will do file exchange and for peers that will do handshake for DHT? Like a protocol muxer on top of a WebSockets connection (or even WebRTC DataChannel directly with the server)? Please do tell me how could I expose better the webrtc-explorer DHT in a way that we could make that experiment :)

[maxogden]

A couple random thoughts:

1. QUIC/UDP DataChannels (https://www.youtube.com/watch?v=mIvyOFu1c1Q) will make a WebRTC DHT a lot more efficient, as it has a 0RTT protocol for p2p connections, vs the 4RTT one used by SCTP/DTLS/UDP Data Channels today
2. Perhaps we can store-and-forward peers external ip:port from the initial STUN and have some mechanism for attempting to have these peers plus new peers hole punch each other without having to coordinate through a STUN server.

The idea here is that once you've holepunched once (see 3.4 here for background), and you have a cone NAT, your public ip:port should stay the same for all future connections.

Let's say you have peers A, B, C. A and B successfully holepunch to each other through a STUN server. A and B now know each others external ip:port. Lets say C comes along and connects to B, also through STUN. B and C now know each others external ip:port.

A <--> B <--> C

To get A and C connected, we could do STUN again, but thats expensive. We can use B to swap SDPs between A and C and have them initiate a holepunching dance without STUN. This would bypass the ICE candidate collection phase (since we are effectively caching it here) and also avoid extra roundtrips to the STUN server.

[maxogden]

According to someone on twitter, if you explicitly set ICE/STUN servers to empty arrays you can skip most external ICE checks (no way to skip internal ones though).

[maxogden]

Update: you can't skip internal ICE, because you need to find out the internal port that was created for your new SCTP session, so you can relay this port to the other peer. Since peerconnections will always have different ports (e.g. they wont do port multiplexing since its an entirely encrypted protocol), my scheme proposed above where you reuse the SDP from an earlier peer won't work because you won't know their new port without directly asking them, which ends up being the same number of round trips as the normal ICE candidate + signaling flow :(

[allouis]

@maxogden Although that flow is the same number of round trips, it seems like a better option because we end up with "signaling peers" rather than servers, keeping the overlay network decentralised.

@feross Do you think having a WebRTC backed dgram module would be a good start to the bittorrent-dht being usable in the browser, or we should be starting this from scratch?

I've been working on some small prototypes for this and would be interested to hear thoughts.

[substack]

I think a dgram overlay network is the most exciting way to tackle this problem because it means we can use bittorrent-dht as-is and we can also implement all sorts of other distributed protocols that were implemented under an assumption of udp/tcp primitives.

Unfortunately, an overlay network with forwarding means we'll need to implement routing algorithms. From what I gather, mesh routing algorithms don't scale very well (perhaps up to hundreds of nodes or maybe low thousands of nodes), so internet-scale routing systems are split up into interconnected autonomous systems (AS) with bridge nodes that advertise the autonomous system prefixes on their local network. I haven't found any research papers so far that bring together all of these ideas into a comprehensive technique, so we might need to experiment with how to glue these ideas together. Here's a good overview of how these routing systems interconnect: http://www.cc.gatech.edu/~traynor/cs3251/f13/slides/lecture13-routing.pdf

As for routing protocols, babel seems like a good candidate. I've already started to implement babel with some simulations to check the results.

Each AS could maintain a minimum number of connections to other AS networks so that it's very unlikely to get a disconnected graph.

Which AS you join on start can be a product of the node ID and then an AS could split when it gets too big.

Here's what I have so far for the babel routing protocol implementation: https://github.com/substack/babel-routing-protocol

[substack]

It looks like IPFS might have a similar approach separating peer routing and content routing to cope with NAT traversal: ipfs/specs#1

[allouis]

we can also implement all sorts of other distributed protocols that were implemented under an assumption of udp/tcp primitives.

^ This would be a great place to get to.

mesh routing algorithms don't scale very well (perhaps up to hundreds of nodes or maybe low thousands of nodes)

This is an issue, but maybe not one we need to address immediately, if I understand Kademlia correctly the number of open connections we'd need to support the dht would be 160 (or whatever number of bits we use for nodeIds), the browsers maximum peer connections is 256 so this seems pretty like it could be an option

What concerns me about a dgram overlay network is how we bootstrap it? Ideally it would function as the other browserify alternatives to core node modules, a drop in replacement with the same api, would we have browser only methods for doing this?

[ghost]

Having looked at nazar-pc code, i have two criticism:

1, Why it's not written in typescript - when i'm looking at the webtorrent code and the dht - lifescript one,
i have a feeling reading a c++ code.
2, WebSocket signaling server has no benefit in the case of DHT.
Websocket is expensive stuff, you cannot imagine that some milions of node would connect to the same webserver, if my assumption is correct looking at the code. (i'm not familiar with lifescript, and to be honest it's very hard to read without classes, single responsibility etc.)

The signaling should be done through the K-RPC protocol, sending the RTCIceCandidates, and storing it in the K-Bucket instead of the IP:port addresses.

I have investigated the technologies in the last few days, and that's what i learnt, or got as an idea, if i understood the whole thing correctly, which is to be honest very hard, processing thousands of javascript code. (webtorrent depends on many)

[nazar-pc]

1. I wrote it in dialect I like, it works like any other JavaScript after compilation, but you need to be familiar with LiveScript in order to read it freely
2. Why no benefit? It seems like a natural fit, bi-directional connection in contrast to HTTP.

you cannot imagine that some milions of node would connect to the same webserver

There is no need to have millions of connections at the same time and even then you can just throw load balancer on it and scale as much as you need. I'd expect WebRTC to be a bottleneck here rather than WebSocket.

I think it would be more appropriate to discuss implementation details in the mentioned repo to avoid distracting lots of people subscribed to this issue. Feel free to open issues there. I'll be using that implementation for one of my future projects that is not WebTorrent-related, so any feedback is definitely appreciated.

[RangerMauve]

sending the RTCIceCandidates, and storing it in the K-Bucket instead of the IP:port addresses.

Sadly you can't reuse SDP or ICE, you can only use it once per PeerConnections, so storing it for later won't make sense.

I think that signaling over k-rpc when possible would be better than always using websockets

[nazar-pc]

I think that signaling over k-rpc when possible would be better than always using websockets

In my implementation WebSocket is only used briefly during bootstrap (as we can't connect other nodes without signaling using WebRTC). As soon as WebRTC connection is established, the rest of signaling is sent using established WebRTC connections with other peers.

https://github.com/nazar-pc/webtorrent-dht/blob/master/bep.rst should explain how this works in details.

Storing SDP for future connections is not possible unfortunately.

[jimmywarting]

Have someone ever thought about using web push?

• You get a token and a endpoint that you can share around to other ppl so they can message you back
• it's persistent like a ip:port (so you can reconnect to same node after you restart or get some network failure)
• you don't need to have a open socket connection to each and everyone all the time
• sdp signaling can be exchanged throught push payload to start establish a new webrtc connection

The bad parts

• Requires service worker
  • can only be installed on whitelisted hosts (https or http://localhost) but i think this is good that everyone is pushing for https
• Must ask for notification permission to enable it (even if you only sending background push)

navigator.serviceWorker.register('sw.js').then(function(reg) {
  return reg.pushManager.subscribe({userVisibleOnly: false})
})
.catch(function(err) {
  // Chrome currently only supports the Push API for subscriptions that
  // will result in user-visible messages. You can indicate this by calling 
  // pushManager.subscribe({userVisibleOnly: true}) instead. 
  // See https://goo.gl/yqv4Q4 for more details.
});

[RangerMauve]

I was actually thinking about this a while ago.

I think the problem I was seeing was that you can't send a push notification to another user from within a browser due to CORS restrictions. Unless the push notification service has the proper headers, the browser won't allow sending POSt requests to it.

[jimmywarting]

due to CORS restrictions

Yea, notice that too. I wonder why they didn't enable CORS. forced to use a backend/proxy server
if i want to build a peer to peer chat between two people that have friended eachother and exchanged token then i think they should be able to ping eachother without having to go throught a backend

Edit: mozilla's endpoint responds with CORS headers

[codebudo]

Some sort of back end is required to tell Peer A where Peer B is (and visa versa). This is the function of a STUN/TURN server for NAT traversal in P2P networks and can be very light weight. Would something else be required to meet CORS requirements?

…

On Nov 30, 2018, at 8:05 AM, Jimmy Wärting ***@***.***> wrote: due to CORS restrictions Yea, notice that too. I wonder why they didn't enable CORS. forced to use a backend/proxy server if i want to build a peer to peer chat between two people that have friended eachother and exchanged token then i think they should be able to ping eachother without having to go throught a backend — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[jimmywarting]

A STUN/TURN server don't have to do any CORS preflight stuff as a client you never send any ajax request to those servers - it's pretty much handled for you in the background

[mikeal]

I should have updated this a long time ago.

I had an informal conversation about 4 months ago with some standards/browser folks that work in this area. From their point of view, a bunch of p2p people have been asking for raw socket access and the ability to open ports in order to resolve this, which is just never going to happen in the browser.

Once we talked through it, I landed on language that was really helpful: What we need is a re-usable signal for WebRTC connections.

For a variety of security reasons the browser can't scrap the signal exchange flow but we could potentially create a signal that is reusable and could be added to a DHT and used by other peers for a longer period of time.

[jimmywarting]

re-usable signal for WebRTC connections.

Are you talking about ORTC? have heard about ORTC but never investigated it.
Is ORTC something diffrent then WebRTC?
How can we have re-usable sdp?
which browser support it?
can the same sdp be used by others?

[RangerMauve]

Edit: Disregard that, I wasn't being constructive.

[mikeal]

I haven't spent enough time with ORTC to know if they solved this or not.

In general, it's best to be a little less specific with browser vendors on feature asks like this. Describing the ask in existing WebRTC terms that don't change the security model except in one specific way which you actually need is the best way to work through a variety of possible solutions.

The performance of data channels makes it very costly to have many connections and doesn't scale well

This isn't a given. I have my own thoughts on why the current performance is terrible but performance is a solvable problem. Nothing about WebRTC makes it inherently less performant and my current view is that the implementations are just old and nobody is really touching them, the browser vendors are just binding to a bunch of RTP libraries.

People complain about the performance a lot but, according to Mozilla, nobody has ever logged a usable issue on the subject. What aspect of performance is bad? Where's a reusable test case?

For the most part, people who write browsers are not also writing web applications, so you can't assume they are aware of things we are aware of unless someone has done the work of properly communicating it to them. Yes, they don't make it easy, they all have obtuse bug tracking processes that are used by literally no other project, but that's the situation we're in.

There's no way to accept incoming connections without signalling servers

If you had a re-usable signal that lasted 24 hours you'd be fine.

[mappum]

What we need is a re-usable signal for WebRTC connections.

Seems like a worthy goal, but to implement re-usable signals wouldn't you need to build a DHT or central routing service anyway? We'd have to assume users are roaming around to different networks and would need to locate them as their address changes.

[nazar-pc]

WebRTC-based DHT is definitely possible and while performance is a concern, potential number of users in this case I think is several magnitudes higher.

I've actually studied DHTs and various approaches to their construction for a while and created another WebRTC-based DHT called Detox DHT (and ES-DHT that is used as generic independent framework under the hood) that I use instead of previously mentioned version.

It is built from scratch using alternative design that takes some major ideas from Kademlia and other papers, but doesn't replicate Mainline DHT with its inherent incompatibilities with WebRTC.

It has somewhat different focus, but still might be of interest to others.

Detox DHT repository contains source code and tests. By following link to ES-DHT it is based on, you can find framework's source code, tests, design document, specification and references to papers.

[mikeal]

Seems like a worthy goal, but to implement re-usable signals wouldn't you need to build a DHT or central routing service anyway? We'd have to assume users are roaming around to different networks and would need to locate them as their address changes.

You have this problem no matter what. DHT's that store IP addresses suffer from the exact same issue.

[mappum]

@mikeal Right, just trying to point out that getting browsers to allow re-usable SDP just pushes the problem of implementing a DHT onto them (which maybe is what we want).

[RangerMauve]

Assuming you had reusable signals, how would connections work?

Would it be something like:

Receiver side:

• Create something similar to a RTCPeerConnection
• Create SDP offer or something similar
• Publish somewhere public
• Listen on incoming connections
• Create RTCPeerConnection per connection
• Somehow configure it with your offer?
• Somehow get their SDP answer?
• Profit!

Initiator side:

• Find the published SDP
• Create an RTCPeerConnection
• Set the SDP offer
• Create SDP Answer
• Somehow get it to the person? This will need centralization in the form of signalling servers
• Profit!

You could potentially have a DHT that bootstrapped by talking to an initial centralized node, and then using this reusable SDP from then on, but you're still constrained by having to rely on centralized signalling servers.

getting browsers to allow re-usable SDP just pushes the problem of implementing a DHT onto them (which maybe is what we want).

I'm 100% into this. A few months ago I was trying to convince the folks behind the Beaker Browser to provide an interface to the DHT used for Dat in order to enable different types of p2p applications. Since it'd be cheaper to have a single DHT set up per browser rather than per browser tab or origin, having a DHT API provided at the browser level helps performance, and having a standard high level API simplifies p2p application development.

This is currently being replaced by a higher-level PeerSocket API that goes a step further by providing an interface that opens sockets to peers for a given channel and abstracts away the DHT or MDNS systems that are behind the scenes.

[klueq]

What could make some progress on this is an elaborate 10-page proposal about what needs to be changed in WebRTC to make this P2P scenario possible. The suggested changes need to be simple and they should also bring some business value. The proposal could make a point about the emerging market of P2P apps or maybe eliminating the ICE step from video calls and thus making them connect faster. Then this proposal need to be presented to the decision makers and if they are convinced, we'll likely see in a year how the WebRTC team rolls out the new API. I would push for an option where a previously established RTCPeerConnection could be saved and then restored by id: WebRTC would send a UDP message to the saved ipv4:port and if the response is correct, we assume that it's safe to restore the connection and skip all the SDP handshakes.

Another option is to let the desktop nodes of webtorrent act as signalling servers.