How does WebTorrent work?

[feross]

Let me try to clear up a little confusion about what WebTorrent is and how it works.

The goal of the project is to build a browser BitTorrent client that requires no install (no plugin/extension/etc.) and interoperates with the regular BitTorrent network, though it uses WebRTC Data Channels for peer-to-peer transport.

To accomplish that goal, I have two objectives.

The first is to develop a working JavaScript BitTorrent client as a Chrome App. This is now finished, though we still need to add support for additional features and polish to bring it on par with other modern clients. I did this first to learn how existing clients work and because lots of this code will be reused by the WebRTC client.

The second objective is to make the BitTorrent protocol work over WebRTC. The protocol obviously needs some modifications, but all the main concepts will remain intact. This new protocol will be called "the WebTorrent protocol", or "WebTorrent extensions".

We'll use this new protocol to ship a standalone JS file that webmasters can add to their site if they want to fetch files over webtorrent. All they need to provide is the info hash (or a .torrent file) (or a magnet uri) and the WebTorrent script will do the rest. We'll have a good default "viewer" UI depending on the file type: PDF viewer, image viewer, audio/video tag (if the format allows for streaming), etc. and also allow the programmer to get the data programmatically without UI too.

This standalone JS file will eventually be the primary WebTorrent client that most users interact with. Eventually, I hope that popular tracker sites will add it to their site to make consuming torrent content easier.

So, why the Chrome App?

We'll add support for the WebTorrent protocol to the Chrome App so it will speak both protocols, meaning it can connect to both normal and web-based clients. This accomplishes two things:

• With a reasonably large install base using the Chrome App, we'll have lots of users to test out the WebTorrent protocol with. We can fine-tune it on real users before we actually ship anything or make any promises about performance. The Chrome App will simply update with better and better WebTorrent support, and they won't have to do a thing.
• The first time a Chrome App client downloads a torrent that no other WebTorrent client is seeding, they become the first WebTorrent seeder. They essentially "bring" the file into the WebTorrent network.

Remember, until mainstream BitTorrent clients support WebTorrent, the web-based clients using the standalone JS file can only download from from other web-based clients. Browsers can't open arbitrary tcp/udp connections -- just WebRTC connections. So, essentially, we're bootstrapping the WebTorrent network by "bridging" it with the existing BitTorrent network.

Eventually, mainstream BitTorrent clients could add support for WebTorrent by using libjingle or other WebRTC bindings (see node-webrtc) so that WebTorrent clients can directly connect to them.

Questions/feedback welcome.

UPDATE: Now planning to use node-webkit to release native binaries for all major platforms instead of a chrome app, due to excessive chrome app CSP restrictions which make development not fun.

[refset]

Rock solid explanation of a rock solid plan +1

[gauravsaini]

I wish I could give this explanation 100000 👍

[eugenioclrc]

+1 i would really want to read something about this

2014-03-07 3:48 GMT-03:00 gauravsaini notifications@github.com:

I wish I could give this explanation 100000 [image: 👍]

Reply to this email directly or view it on GitHubhttps://github.com//issues/39#issuecomment-36972120
.

May the source be with you.

[ChrisMorrisOrg]

Sounds fantastic!

[panuhorsmalahti]

Chrome-only is not a good starting point..

[feross]

Planning to switch to node-webkit so webtorrent can be native on all
platforms without a dependency on Chrome.

Thanks for the feedback.

On Thursday, March 27, 2014, Panu Horsmalahti notifications@github.com
wrote:

Chrome-only is not a good starting point..

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/39#issuecomment-38825976
.

Feross
✩ blog http://feross.org/ | ✎ studynotes http://www.apstudynotes.org/ |☮
webtorrent http://webtorrent.io/

[modeswitch]

@feross is the reason you're using node-webkit because of window/mac support that's missing from node-webrtc?

[yyx990803]

Hmm when you switch to node-webkit, maybe consider using Vue.js for the interface since there's no more CSP issues? ;)

[feross]

@modeswitch no, the reason is that I want to take webtorrent, which is a whole bunch of npm modules, wrap it in a UI, and create downloadable binaries for all the major platforms. I think I could use either the webrtc from node-webrtc or the built-in support in node-webkit and it shouldn't make a difference.

[feross]

@yyx990803 will seriously consider it!

[ivantodorovich]

Is it already working for (Regular)Torrents ?

I spend a lot of time refactoring (and understanding) torrent-stream's and then I find this! It seems I wasted my time doing the same thing as you did :S

I'm interested in a BitTorrent module, similar to torrent-stream, don't really need the WebTorrent stuff. I need it to be fast and efficient. I intended to continue the work of torrent-stream but it seems yours is more advanced.

[ivantodorovich]

In fact, after a deep look into the files I think https://github.com/feross/bittorrent-client is what I'm interest in, really. But its an empty repo.

You are working solely on this repo, intending to split code later on?
Is progress being pushed here, or are you working on your local clone?

I want to help. I promise I'll behave and absolutely not a single semicolon. 👯

[feross]

@ivantodorovich I had torrent downloading working when this was a chrome app, but I'm currently refactoring most of the logic out into bittorrent-client and adding streaming logic, ala torrent-stream. Then, this repo (webtorrent) will simply host a node module and command line app. I haven't pushed my code up to the bittorrent-client repo yet, but I'll do that soon. Totally happy to have your help and contributions!

The reason I can't just re-use torrent-stream is that I need to support multiple torrents without creating multiple dhts (though it would be possible to fix that in torrent-stream) and I need multiple torrent swarms to be able to listen on the same tcp port (so I'll be using my own bittorrent-swarm), and I'll probably be making other design decisions that make building a hybrid bittorrent/webtorrent client easier.

But I don't plan to put any webtorrent protocol specific stuff into bittorrent-client -- that will go into another module. So you don't have to worry about any webtorrent protocol stuff bloating or complicating the codebase.

[feross]

@ivantodorovich I just pushed some code to bittorrent-client if you want to take a look. It's incomplete and untested, but if you want to take a look it's there. I'll continue updating it until it works...

[ivantodorovich]

Great! Im currently at work and I have a busy day ahead. I ll get into it tomorrow.

Take a look at my fork https://github.com/ivantodorovich/torrent-stream/tree/refactor
(PR mafintosh/torrent-stream#25)

Its a working version of torrent-stream, with a refactored code design. Its a lot easier to read than mafintosh's.

[feross]

A few weeks ago, I gave a talk in Budapest where I shared the current state of WebTorrent. There's been quite a bit of progress since things got started in January.

"Bringing BitTorrent to the Web"
https://www.youtube.com/watch?v=PT8s_IVWDgw

If you haven't been following too closely and you're looking for an update, then this is the talk for you. I also spent some time talking about browserify, the npm way, and WebRTC 😄

[cadesalaberry]

Awesome !!!!

What software did you use for your presentation ? It looks gorgeous !

[feross]

Thanks! I used Keynote, which has really nice built-in animations. And then I just came up with a nice color scheme to use. Glad you liked it!

[patcito]

Hey @feross, node-webkit is nice but it won't work on chromeos, so by switching to node-webkit, you'll actually be supporting one OS less which is too bad for us chromeos users I guess. Will there be a way to still run webtorrent in a chrome app? Thanks.

[feross]

Yep, Chrome OS support is still planned. Check out this repo: https://github.com/feross/webtorrent-chrome which has a basic working version. Development effort is focused on the node-webkit version for now but once that's released I plan to bring the Chrome App back up-to-date with all the recent progress :)

[chuckwh]

I'm not a contributor at all, just a watcher, but I added a Sulia post to
help promote (in my little way) Feross' youtube:

http://sulia.com/mixertopics/

Please forgive the blatant nature of the Sulia identity. It's designed as a
promotional thing for a website that uses webRTC and somehow morphed into a
webdev thing and I haven't had time to transition my Sulia identities
properly.

On Sat, May 17, 2014 at 6:39 PM, Feross Aboukhadijeh <
notifications@github.com> wrote:

Yep, Chrome OS support is still planned. Check out this repo:
https://github.com/feross/webtorrent-chrome which has a basic working
version. Development effort is focused on the node-webkit version for now
but once that's released I plan to bring the Chrome App back up-to-date
with all the recent progress :)

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/39#issuecomment-43427199
.

[feross]

WebTorrent is now usable as a command-line program, installable via npm. Give it a shot!

$ npm install webtorrent -g
$ webtorrent --help

Here's how you download a torrent:

$ webtorrent "magnet:?xt=urn:btih:d2474e86c95b19b8bcfdb92bc12c9d44667cfa36"
$ cd /tmp/webtorrent  # this is where all files currently go

Please report any bugs you find! Huge shoutout to @fisch0920 who helped out a ton this past week. Remember, this is just stage 1 of the greater webtorrent project. Next up is:

• more robustness (this is where you can help! please report bugs!)
• desktop apps for mac, windows, linux
• webrtc implementation

Join us in IRC on freenode at #webtorrent if you want to help with development, or you just want to hang out with some cool p2p / mad science hackers :)

[Ayms]

@feross So, if I summarize: you get a chrome app client + node-webkit client (using node-webkit embedded webrtc since as far as I understand node-webrtc is somewhere far from being usable on windows/mac (@modeswitch ?) too bad because I was looking for a node-webrtc like solution), they bring the files to the webtorrent network.

Then you get the webtorrent client inside browsers, this one can discuss using webrtc with the above clients and other browsers clients.

Now, what's the mechanism to really bring a file inside webtorrent? ie using the client browser how can I tell a chrome/node-webkit client to bring the file?

This can raise some other issues like chrome/node-webkit clients seeding some "unauthorized" files, so unloading the risks to them.

That's why the concepts of Peersm clients (https://github.com/Ayms/node-Tor/tree/master/install#peersm-client-installation) which are bridging the content with bittorrent are a bit different, if something can not be found in peersm network, they are asked to find it in bittorrent, then they just relay the data, don't store anything and are as discrete as possible, webtorrent does not have the same anonymity constraints but maybe this thread where the behavior is detailed can be of some interest http://lists.zooko.com/pipermail/p2p-hackers/2014-June/003272.html

[nickdesaulniers]

Hey everyone, cool repo. A coworker, @marco-c, and I seriously discussed doing something like this for Firefox OS about a year ago. Just thought I'd share some feedback based on our discussion.

I like the statement: We'll use this new protocol to ship a standalone JS file that webmasters can add to their site if they want to fetch files over webtorrent. I think that is a concise and clear goal to work towards. If any site could include such a library as easily as they could jQuery, this would have tremendous effects. Unfortunately, releasing a chrome extension is marginalizing users to Chrome or Chrome OS (can you run Chrome extensions in Chrome for Android?) and a node-webkit client is marginalizing users to Windows, OSX, or Linux (cutting support for Chrome OS, Firefox OS, Windows phone, blackberry, game consoles, etc as pointed out by @patcito ). And frankly, if I have to download a binary, what makes this project different from any other BitTorrent client other than the UI being written with HTML, CSS, & JS?

The second issue, that was a blocker for us at the time, was interop with existing torrent trackers. The torrent trackers all seem to be using UDP, which you know is a limitation to current browsers. Firefox OS recently landed support for UDP sockets, but we just covered why that solution is less than optimal. UDP sockets are being spec'ed out as part of the Raw Sockets proposal, but are only implemented today in FxOS. And I'm wouldn't hold my breath on adoption by other browser vendors or even desktop FF, but how else could we ship an email client based solely on web tech in FxOS?

The third issue: interop with existing BitTorrent clients. I think you addressed this concern accurately: Eventually, mainstream BitTorrent clients could add support for WebTorrent by using libjingle or other WebRTC bindings (see node-webrtc) so that WebTorrent clients can directly connect to them.. The biggest issue is that until bittorrent clients pick up the numerous WebRTC protocols, someone is going to have to run what is essentially a proxy server to proxy messages from WebRTC dTLS to UDP. That's both expensive, and I wouldn't trust such a service.

The final issue: only Chrome implements the FileSystem API. This is kind of necessary to support the ability to seed files, especially when the tab is reloaded. For example, let's say I have a bunch of input type=file tags in my DOM. WebRTC is using data channels to share the array buffer backing each file. Ok, great, I'm seeding all these files now. Then I command + shift + r and all of a sudden my inputs are cleared. I would literally have to re add every single file that I want to seed again! That kind of UX isn't going to get you very far.

Now that I think about it, I would recommend writing such files to indexedDB, then in the client, have UX for allowing the ArrayBuffer/File/Blob to be read out of indexedDB and downloaded to the host's filesystem. Though indexedDB is my least favorite API, and isn't available everywhere, there's libraries that help with that.

As always, don't let those who say it can't be done get in the way of those doing it. ;) Keep up the good work! Always happy to help, or host a hacking session at Mozilla in Mountain View.

[modeswitch]

@Ayms node-webrtc has darwin support, but Windows is not ready yet. There's no technical reason why it can't run on Windows though, mostly lack of contributions for that platform.

@nickdesaulniers I'll pipe up and point to https://github.com/js-platform/filer/ (disclaimer: my project) as a file system layer on indexeddb. It implements Node's fs API, so it's familiar and easier to use than raw indexeddb.

I'm also happy to host webrtc hacking sessions at Mozilla Toronto if there's interest.

[Ayms]

@modeswitch , thanks, when it's time I will definitely look at a node based client supporting webrtc for the Peersm clients (because in Peersm context some are quite allergic to google things, I don't know if there are plans at Mozilla for something like a ff app ideally working on any platform, including ff os), node-webrtc if applicable, unfortunately for windows stuff I am afraid I am not the right person to be able to contribute.

@nickdesaulniers , sharing some thoughts too since some Peersm concepts are close to Webtorrent.

As far as I see it the webrtc clients (that we call Peersm clients in our case) are just helpers to bring the content and maintain it alive (ie bridging to bittorrent) if too many peers close their browsers, a bit like the solid seeders in bittorrent, indeed some need to load a binary which does not make it very different from a standard client but that's supposed to be a small part of the users.

Another point is that you are offloading the risks to that clients, that's why Peersm clients are implementing https://github.com/Ayms/torrent-live, they totally freeride, hide what they are really requesting and block detected spies, again that's supposed to be marginal, if not they would screw up the bittorrent protocol and everybody would end up in the blocklist.

Regarding trackers or DHT, this could become an issue the day (if it happens) bittorrent clients talk webrtc (because in the meantime you know browsers can not talk to them and then implement a separate DHT), but then the DHT will talk webrtc, probably not trackers but we don't care I think.

Up to now I did not really study the need for a FileSystem API, since it's not available, or let's say the current behavior using indexedDB seems enough (and probably something like filer on top of it would make it better), despite of some problems like clearing everything in Chrome if you clear the cache and some bugs reported to FF of corrupted database, these "small" drawbacks are of course not cool at all if you had dozen of GB files stored and then you must reupload everything inside your browser, let's assume this will improve in the future

I am focusing more on the issue of all W3C API not handling delta data (File, indexedDB, etc), I can provide some links if there is an interest, that's really painful and inefficient, as well as the lack of streams support, which will solve the delta data issue too, we know Streams are coming but I find the process really too long and most of concerned APIs seem not to care a lot.

[feross]

@nickdesaulniers Thanks for your detailed feedback and questions! I've tried my best to answer your questions inline.

Ship a standalone JS file that webmasters can add to their site if they want to fetch files over webtorrent.

This is the goal. And it's now complete! WebTorrent now works in the browser, end-to-end!

I deployed an example app that uses WebTorrent here: http://instant.io The code is really simple -- check it out: https://github.com/feross/instant.io/blob/master/client/index.js

We need better docs, a getting started tutorial, a pretty website (#29), and more. Once we have that, we can officially announce an alpha release!

if I have to download a binary, what makes this project different from any other BitTorrent client other than the UI being written with HTML, CSS, & JS?

Correct. The main reason to build a native bitorrent+webtorrent app is to prove that it's possible for "hybrid" clients to work well -- and to get some clients out in the wild. The long-term goal is to get the popular bittorrent clients on the market today (uTorrent, Transmission, Vuze, etc.) to add WebRTC support, bridging the two worlds -- web and native.

interop with existing torrent trackers

You're right. We can't use UDP because that's not allowed from sandboxed webpages. But there's another, more serious issue with this approach: the WebRTC signaling process requires an offer/answer to be sent between two peers. Thus, the server needs to ability to push data down to the offering peer once the other peer sends back an answer.

To solve both problems, we've modified the tracker protocol in the smallest way possible to include websocket support. The code for that is here: https://github.com/feross/webtorrent-tracker. WebSockets won't scale as well as UDP trackers do, but this approach works today! Feedback is welcome of course -- none of this is set in stone yet.

The long-term goal is to build a WebRTC DHT and be minimally reliant on trackers.

interop with existing BitTorrent clients. ... The biggest issue is that until bittorrent clients pick up the numerous WebRTC protocols, someone is going to have to run what is essentially a proxy server

It's not a good idea to implement any sort of artificial proxy bridge between WebTorrent and BitTorrent, even as a stop-gap solution until native clients add WebRTC support. No one wants that liability (anything could be transfered) and it's expensive.

Even if the two networks are not joined from day 1, WebTorrent is still quite useful on it's own. For example, file sending services can be built. See: http://instant.io (source code: https://github.com/feross/instant.io) Instant uses torrent swarming for super efficient downloads. The UX needs a bit of work, but the potential is there. And there are many use cases that we likely haven't even thought of yet!

Another point is that it doesn't take many people running the hybrid client for the network to be pretty robust. There is so much excess capacity on modern torrents because of super fast internet connections these days and long-term seeders.

Filesystem

I think IndexedDB will work, but we should consider all possible approaches. There are many excellent modules available to make that API nicer to work with. I opened a separate issue here (#86) to solicit feedback.

A WebTorrent / WebRTC / P2P hackathon at Mozilla in Mountain View sounds excellent. I can help organize if that's useful. Hit me up at feross@feross.org and let's make this happen!

[Ayms]

The long-term goal is to build a WebRTC DHT and be minimally reliant on trackers.

I have started to write something about the "WebRTC DHT" starting here: https://github.com/Ayms/node-Tor#dht-and-bridges , maybe it's a little bit too much summarized for now, I will expand when I have time.

For WebTorrent, the "modulus" does not apply and the "bridges" are the trackers, but for Peersm the bridges are only keeping references to peers, not to the content which is only referenced in the WebRTC DHT.

The goal is of course to minimize the role of the trackers/bridges (and persistent WebSockets connections to them) for peers discovery and introduction.

I think both projects need the same WebRTC DHT concepts, should we open a new issue to track this (and a gist to spec this)?

[jure]

I think @tsujio's https://github.com/tsujio/webrtc-chord is a good implementation of a DHT (Chord) with WebRTC. It's the only working/complete implementation I found.

[Ayms]

@jure thanks, I have read tsujio/webrtc-chord#2 too.

What I would like to have is a "serverless WebRTC DHT", "serverless" as far as possible, in my case the bridges (https://github.com/Ayms/node-Tor/tree/master/install#node-tor-bridge-websocket-server-installation right now they implement WebSockets and will implement WebRTC later) can relay the introduction when necessary, they are "servers" or peers backend processes, question of wording, but the DHT should favor peers introduction and I don't see it really in the webrtc-chord repo.

[marcus7777]

I'm very excited about this project, what i'd love to build and i think this might make it possible, is a torrent sync version of websites, that is with 2 keys one to edit the website and when to access it. see the website did exist totally peer to peer. maybe having be editable key locked away inside itself, so that people can log in an edit it just like it CMS.

[miketamis]

@marcus7777 problem with "torrent sync" is its imposible edit a torrent because even the slightest change will change the infohash and therefore create a completely new swarm. you could get around this of course by creating a new torrent each time you change the file and telling all the peers in this swarm about this new torrent and then have them redownload the files (using the old files to speed it up) but this becomes very complecated.

I think the first thing todo would be to create a twitter replacement using webtorrent similar to http://twister.net.co/ this way you could see what works and doesnt. or a simple site that allows people to have html pages eg p2ppages.com/ and once you upload a page it can not be change. but you can have say p2ppages.com/saspiron which would link to a different infohash page depending on what i set it to be sorta giving the ablity to edit a page by replacing the old one.

[miketamis]

when i get some time i might work on the p2ppages idea sounds really cool now i think about it :D

[miketamis]

here is a webpage that gets loaded from a torrent http://requirebin.com/embed?gist=00a5d23120b8bafed6b4 therefore it may not ever load i will try seed as much as possible but can garentee the site will ever load. code can be seen here https://gist.github.com/miketamis/00a5d23120b8bafed6b4 and if you want to download the html page that is loaded by this webpage you can do so using instant http://instant.io/#3a3cca2992987699def85776f58c3c510bc2b166

Please message me if you get a Hello World page :D

[yipperr]

@miketamis just loading for me no hello world

[marcus7777]

This is wonderful to see thanks @miketamis what i was thinking of a
'torrant sync' to localstorage from js. There is an open source version of
torrent sync called syncthing it's written in go. I'm just checking help
with that go compiles to javascript.
On 8 Nov 2014 05:51, "miketamis" notifications@github.com wrote:

here is a webpage that gets loaded from a torrent
http://requirebin.com/embed?gist=00a5d23120b8bafed6b4 therefore it may
not ever load i will try seed as much as possible but can garentee the site
will ever load. code can be seen here
https://gist.github.com/miketamis/00a5d23120b8bafed6b4 and if you want to
download the html page using instant you can do so with
http://instant.io/#3a3cca2992987699def85776f58c3c510bc2b166

—
Reply to this email directly or view it on GitHub
#39 (comment).

[miketamis]

Thinking about it you dont wont something like btsync or syncthing because theres a bit of overhead what you want is to store each page in a new torrent and have a logical link (maintained by mabye a DHT or a blockchain) that link to the infohash. html files are so small we can afford to hardfork them as such and we dont need to do any sort of diff patches. each image file will be a seperate torrent. this adds the additional benifit that each site wont need there own copy of a particular image they can just share it (wow that would bring a couple new way to think of image hosting)

[marcus7777]

that sounds like an awesome site. so it would be a torrent for every static file and when you make changes. Replace it. So how does the update happened? i'd love it if we could avoid any centralisation. Could the logical link be held in page itself?

[miketamis]

for each logical link you would generate public and private key. then you would have a DHT where the key for the DHT would be the hash of the public key (this hash would also be consider the logical address) then to attach a logical address (public key hash) to the physical address (infohash) you would sign the infohash and the time and push it to the DHT. the newest signature attached to DHT is the current link as such. I hope this makes sense. This is also an idea (my first one) Ill probable find out something horribley wrong with it.

also futher convo about this idea should be discuss somewhere else.

[feross]

@miketamis neat experiment – thanks for sharing!

[urbien]

@miketamis @marcus7777 - this topic is very interesting to build completely server-less apps. Our app today is bootstrapping assets from the server and then operates serverless on top of bitcoin's blockchain and webtorrent, thanks @feross and everyone who works on this project! But we would love to get app assets from the DHT as well. See that Ethereum's Swarm project is solving this problem as well. Would you like to discuss this offline?
email me: gene dot vayngrib at tradle dot io

[Nashorn]

I would like someone to clarify. In the webtorrent runtime, when someone downloads a .torrent file or makes one for a new file they plan to seed, will the torrent be lost after the session/browser is closed?

[feross]

@Nashorn Yes, some peer needs to be seeding a file for any peers to be able to download it.