Implement Web (HTTP) Seeding (BEP17+BEP19)

[transitive-bullshit]

Web / HTTP seeding is an extension to standard bittorrent p2p which allows a client to download ranges from normal HTTP servers in addition to standard bittorrent peers. This has the advantage of effectively adding permanent, seed-only peers to any swarm, and it's very easy for any existing HTTP fileservers to facilitate bittorrent and also reduce their own load by offloading portions of normal HTTP downloads to bittorrent peers.

I think this feature will be particularly important to the browser version, since we're expecting the number of hybrid peers to be relatively low compared with normal bittorrent clients, and HTTP-based peers will be supported in the browser without any workarounds.

Note that there are two competing standards for HTTP seeding, both with their own pros / cons, and I believe most clients implement them both since neither is very complex. The nomenclature used to refer to the standards is very inconsistent, so for the purposes of webtorrent, I propose we refer them simply as BEP19 (aka GetRight-style http/ftp seeding) and BEP17 (aka Hoffman-style, http seeding, webseeding).

BEP19 has the advantage of being able to work with any file accessible via normal HTTP or FTP, utilizing standard HTTP range requests to download pieces (or better yet reserved spans of pieces to minimize separate HTTP requests). BEP17 requires external servers to be modified to support BEP17, which is a disadvantage in terms of interoperability, but it does have the advantage of allowing the seeding servers to better control traffic and avoid bandwidth abuse.

This task will require modifying parse-torrent to recognize HTTP seed servers. It will also require changes to bittorrent-swarm, possibly bittorrent-protocol, and bittorrent-client if we want to support more efficient utilization of HTTP by reserving spans of pieces for each individual HTTP seed request as opposed to treating HTTP seed peers as normal peers and requesting single blocks at a time from them.

Resources:

• High level overview of HTTP seeding and protocols
• Vuze wiki's overview of its HTTP seeding implementation

[feross]

Thanks for the summary. Agreed - this is super important for webtorrent.js. It's also exciting because it enables a CDN-like use-case for sites that I'd like to support like Internet Archive. They can enable webseed for everything, since they're already doing that. But when other visitors are online, then torrenting will happen :)

One difference for webtorrent.js is that the webseeder will need to enable CORS for two reasons:

• we're requesting data using XHR (to get at the actual bytes) and that requires CORS
• we're sending a custom header (Range) and that requires CORS

[thomasdelaet]

Is anybody working on this feature? Would somebody with a bit more experience in the code base be available for support when I try to implement this?

[trevorstarick]

As in DHT?

[transitive-bullshit]

@trevorstarick web seeding is unrelated to DHT.

@thomasdelaet nobody's currently working on this, but we welcome any help :) I'd be happy to help you with any questions regarding an implementation. Either reply to this thread with specific questions or if you'd like to chat, feel free to reach out at fisch0920@gmail.com

[trevorstarick]

Is a JavaScript implementation of DHT still needed/wanted. I'm interested to help but not sure what or where to start.

[transitive-bullshit]

There is an existing DHT module used by webtorrent, bittorrent-dht, though it definitely still needs improvement.

[macbroadcast]

Isnt most of it build allready in the the https://git.popcorntime.io/stash/projects repo ?

[transitive-bullshit]

@macbroadcast the webtorrent project shares a few modules with popcorntime's underlying torrent engine (peerflix / torrent-stream) but is fundamentally different because it's aim is to work in the browser as well as in node. This thread is for one particular issue within the greater webtorrent project, and we're getting off-topic, so check out webtorrent.io for further details.

[transitive-bullshit]

Hmm I wasn't aware popcorn-time had moved away from using peerflix. Their httpstream could be useful.

[jure]

Sorry for the slightly off-topic comment, but @fisch0920 and @trevorstarick who were asking about JS DHT, you should check out: https://github.com/tsujio/webrtc-chord

[miketamis]

I have started to work on this.

[feross]

Awesome! You should join us on IRC at #webtorrent (freenode).

[feross]

A generous individual contacted me to express his interest in this feature. He's willing to put up $500 to see this feature finished.

@miketamis What's the status of your work on web seeding? This would be a nice bonus for finishing a feature you were already working on! :)

[miketamis]

Situation is good, HTTP seeding is working (currently only BEP19 the one that supports standard HTTP) just currently refactoring webtorrent to make the addition more modular.

[feross]

@miketamis can you send a PR for web seed support, or send a link to your in-progress work on web seeding? I'd like to take a look at it.

[espoal]

any news?

[FluorescentHallucinogen]

FTP seeding is described in BEP19.

GetRight supports FTP seeding.

The browser can download at any point (or at the beginning at worst case) of the file via FTP and seed (upload) via BitTorrent. Are there any technical limitations to not implement FTP support in WebTorrent?

[transitive-bullshit]

My bad; it's been awhile since I've looked at them :)

[olalonde]

I don't think it's possible to do FTP in the browser so that would be a Node.js only feature if it gets implemented.

[rom1504]

it's quite possible. try this ftp://speedtest.tele2.net/

[olalonde]

AFAIK some browsers double as an FTP client but there is no standard way to download FTP files from Javascript.

[DiegoRBaquero]

@FluorescentHallucinogen In node it would be possible. It would just need to be implemented. But I don't see why not use HTTP instead.

In the browser it is not possible, FTP doesn't have cors and XHR (AJAX) doesn't support it. See this

[FluorescentHallucinogen]

Are there any differences between &as= and &ws= parameters in magnet links? Is &as= an alias of &ws= in every respects?

I know that &ws= parameter according to the specification supports links to directories (folders) for multi-file torrents.

Does &as= parameter according to its specification supports links to directories (folders) or only to single files?

[FluorescentHallucinogen]

If one of the trackers in the magnet URI e.g. magnet:?xt=urn:btih:df2a06c6d602f5514f11246dcb843efa403efe0c&dn=amplitude-new-zealand-4k.mp4&tr=ws%3A%2F%2Ftracker.fastcast.nz&tr=wss%3A%2F%2Ftracker.btorrent.xyz&tr=wss%3A%2F%2Ftracker.openwebtorrent.com&tr=wss%3A%2F%2Ftracker.webtorrent.io&ws=http%3A%2F%2Ffastcast.nz%2Fdownloads%2Famplitude-new-zealand-4k.mp4&ws=https%3A%2F%2Fwebseed.btorrent.xyz%2Famplitude-new-zealand-4k.mp4 is insecure (starting 'ws://') e.g ws://tracker.fastcast.nz the whole process of downloading torrent from the page loaded over HTTPS e.g. https://instant.io or https://btorrent.xyz is blocked by the browser. Connecting to the rest of secure trackers from the magnet URI (starting wss://) e.g. wss://tracker.btorrent.xyz, wss://tracker.openwebtorrent.com, wss://tracker.webtorrent.io does not occur.

Is it possible to implement in WebTorrent avoiding connecting to a non-secure WebSocket server from a secure origin?

[sirajea]

@FluorescentHallucinogen There is very little documentation. I found https://wiki.theory.org/BitTorrent_Magnet-URI_Webseeding and https://en.wikipedia.org/wiki/Magnet_URI_scheme#Web_links_to_the_file
AS stands for acceptable source (fall-back source), XS for exact source. WS stands for web source/ web seed? The parameters can also be used to retrieve metadata (the .torrent file) instead of the actual payload. Webtorrent's magnet-uri doesn't seem to be using the exact source https://github.com/feross/magnet-uri/blob/master/index.js#L79-L85 and WS overrides AS.

[feross]

@FluorescentHallucinogen the issue you described with insecure ws:// trackers crashing the client on secure https:// origins is now fixed, as of bittorrent-tracker 7.5.2.

[FluorescentHallucinogen]

AS stands for acceptable source (fall-back source), XS for exact source. WS stands for web source/ web seed? The parameters can also be used to retrieve metadata (the .torrent file) instead of the actual payload.

Does it mean that it is possible to use &as= and/or &xs= and/or &ws= as a link to *.torrent file in the magnet URIs?

For example:

magnet:?xt=urn:btih:6a9759bffd5c0af65319979fb7832189f4f3c35d &as=https%3A%2F%2Fwebtorrent.io%2Ftorrents%2Fsintel.torrent

and/or

magnet:?xt=urn:btih:6a9759bffd5c0af65319979fb7832189f4f3c35d &xs=https%3A%2F%2Fwebtorrent.io%2Ftorrents%2Fsintel.torrent

and/or

magnet:?xt=urn:btih:6a9759bffd5c0af65319979fb7832189f4f3c35d &ws=https%3A%2F%2Fwebtorrent.io%2Ftorrents%2Fsintel.torrent

Does WebTorrent support using &as= and/or &xs= and/or &ws= as a link to *.torrent file?

[feross]

@FluorescentHallucinogen You should use &ws=.

[feross]

Closing this now, as multi-file web seeds were the last thing we needed to support web seeding fully in WebTorrent, and that is now supported as of this PR (https://github.com/feross/bittorrent-swarm/pull/34). Note that we chose to support BEP19 and not BEP17, since the former is more web-friendly. I haven't seen any torrents that use the BEP17 style.

[FluorescentHallucinogen]

You should use &ws=.

@feross Do you mean &ws=https%3A%2F%2Fwebtorrent.io%2Ftorrents%2Fsintel.torrent?

[feross]

Sorry for the confusion. You should use &ws= to specify the http URL where the files in the torrent are. See this spec: http://www.bittorrent.org/beps/bep_0019.html

We don't support specifying the torrent file location in magnet links at the moment, but I recently discovered this spec (https://wiki.theory.org/BitTorrent_Magnet-URI_Webseeding) that explains how to interpret all these parameters correctly.

There's a new issue here that will track this feature: #714

[JesseWeinstein]

I'm looking into supporting WebTorrent at the Internet Archive, and ran into a (minor) missing feature in the support for this. Specifically, it appears that WebTorrent doesn't respect the "name" field in .torrent files, which means it looks in the wrong place when trying to load multi-file webseeds that define the "name" field.

https://wiki.theory.org/BitTorrentSpecification#Info_in_Multiple_File_Mode -- definition of the "name" field
https://archive.org/download/mbid-3124c085-4665-429b-a188-bef1d8aa0a25/mbid-3124c085-4665-429b-a188-bef1d8aa0a25_archive.torrent -- example of a torrent that uses it (and webseeds).

(edit: After further examination, it looks like WebTorrent does use the name field for looking up webseeds; instant.io and btorrent.xyz just don't display their support for it)

[feross]

@JesseWeinstein It's awesome that you're investigating supporting WebTorrent at The Internet Archive! That's been one of my primary goals in developing WebTorrent from the very beginning! Do let me know if there's any way that I can help.

So, about the web seed issue you saw. @dcposch added support for web seeds about a month ago, but it looks like instant.io hasn't been redeployed since then. I just pushed the latest code to it. Let me know if it's working for you now.

You might need to run a local copy of Instant.io since the production site uses Strict Transport Security to force HTTPS and that causes HTTP web seeds to fail with Mixed Content. This is a use case that the standards bodies have not considered. A secure HTTPS application that knows what content it is expecting should be allowed to request data over HTTP.

[KrishnaPG]

Looking at web-seeding - is the https://webtorrent.io using it?

The reason I am asking is, on the site's homepage there is a sintel torrent file being downloaded. Now, to test it, I have opened the same site in multiple tabs on the browser.

What I was expecting is, that the new tabs will automatically pickup the previous open tabs as (local) seeders for the content - but that did not happen. Every new tab (page instance) tried to hookup for its own peers from outside, essentially downloading multiple copies of the movie from scratch from same peers.

Theoritically,

1. each loaded page instance should become a seeder capable of serving other local page load requests (from other tabs or browser instances)
2. the local page instances (being on the same machine) should become some kind of "local swarm" that partition the remaining work into different chunks and each dowload exclusive portions separately from the "global swarm" and once done spread them equally locally.

For example, as of now 3 tab instances are downloading bytes 0 to 120MB all 3 times separately (without any knowledge of local peers doing the same thing).

What I would love to see is: 3 tabs each divide the remaining job into 3 pieces (0 to 60MB, 61to 90MB, 91 to 120MB etc.) and each connect to entirely different set of peers and finally merge the content locally. If a new tab opens it should reuse the old content from others are they all re-divide their work (based on remaining portion and completion status)

Is this kind of thing possible with WebTorrent as of now?

[DiegoRBaquero]

@KrishnaPG

Looking at web-seeding - is the https://webtorrent.io using it?

Yes, it is, but everything is downloaded to memory, no persistent chunk store is being used at this moment.

What I was expecting is, that the new tabs will automatically pickup the previous open tabs as (local) seeders for the content

That would depend more on the tracker, but it'd would definitely be nice. I've opened an issue here.

What I would love to see is: 3 tabs each divide the remaining job into 3 pieces (0 to 60MB, 61to 90MB, 91 to 120MB etc.) and each connect to entirely different set of peers and finally merge the content locally. If a new tab opens it should reuse the old content from others are they all re-divide their work (based on remaining portion and completion status)

That would require syncing information between the 3 tabs. Although it can be done, it's not on the scope of WebTorrent itself. What should be done is connecting the 3 tabs between each other so they can share the pieces they already have at local-link speed. (This already happens when you are the only peer)

[feross]

@KrishnaPG What @DiegoRBaquero said.

Also, web seeds are supported fully now, so we can finally close this issue.

[KrishnaPG]

Thank you. Just to double confirm, each web-browser instance acts as a potential seed for any new near-by clients, correct?

Which means, if 10 people from near-by location connect to one web-site, any new further connections from near-by locations have the chance of picking up the content from one of these previous browser instances (rather than from the server directly) - correct? Kind of browser-based peer-to-peer CDN

[feross]

@KrishnaPG Yes, torrents + web seeds are a form of peer-assisted CDN.

But physical proximity (i.e. nearby peers) has nothing to do with how peers discover each other. The tracker server gives out peers randomly, and the client keeps peers that are fast and cooperating. The http server (web seed) is treated as a peer too, albeit a really good one.

[lock]

This thread has been automatically locked because it has not had recent activity. To discuss futher, please open a new issue.