Webtorrent v0.105.0 won't download from webseed if no peer online

[ziro-official]

What version of WebTorrent?
v0.105.0 and v0.99.0
What operating system and Node.js version?
Ubuntu 18.04
What browser and version? (if using WebTorrent in the browser)
No
What did you expect to happen?
v0.105.0 won't start download from webseed but v0.99.0 did.
What actually happened?
When I create a client from a .torrent file, Webtorrent won't start download from webseed if no peer online (fresh torrent, no one seed it, no metadata on any tracker). I tried to uninstall v0.105.0 and install v0.99.0 and it work, it start download from webseed right away.

[SilentBot1]

Hi @ziro-official

Please could you provide a code example of how you are attempting to achieve this in v0.105.0?

Kind regards,
Brad

[feross]

We need a code example to attempt to reproduce this. Thanks!

[jimmywarting]

Think i have stumble upon this to before.
think it's was when you only have a magnet link and not the full torrent metadata?
the magnet link may contain one ws (webseed) without a xs (exact source to torrent file)

but you say that you already have the torrent file/metadata

if there where nobody downloading this and xs where not attached it would not download anything? (guessing)

magnet:?xt=urn:btih:08ada5a7a6183aae1e09d831df6748d566095a10
&ws=https://webtorrent.io/torrents/Sintel.torrent # folder to all the files
&xs=https://webtorrent.io/torrents/sintel.torrent # actual torrent file

[jimmywarting]

While i was looking at the sintel.torrent source i could not find any information regarding a xs in the torrent file?
That would result in a magnet link not having any xs when they are generated

[SilentBot1]

@jimmywarting that's what I was thinking.

I just wanted to double check with @ziro-official how they were adding the torrent, as if they were adding it via the MagnetURI without &xs or peers, this would be causing the torrent to not load as no metadata is available.

The sintel.torrent downloaded from WebTorrent.io doesn't include a xs location as the full torrent buffer is embedded into the JavaScript itself, meaning if a MagnetURI was generated from this .torrent, it would contain no xs location.

If @ziro-official is adding the torrent with the .torrent file, this would be an unrelated issue than what we're discussing, which is why I asked for more clarification.

[PaTiToMaSteR]

I hope this helps. This is how today I got it working.

1. In the server I have a folder called torrents
   1.1 .../torrents/folder_with_video_and_other_stuff
   1.2 .../torrents/folder_with_video_and_other_stuff.torrent
2. I used WebTorrent to create the torrent, drag and drop the folder, remove all trackers (still inserting them... doesn't matter what you do it's hardcoded)
   2.1 Add my own tracker address: ws://IP:8000
   2.2 Mark it as private
   3 Allow in your website to list the folder for web seeding.
   3.1 I edited on my apache virtual host config as:
   <Directory "/home/USER/public_html/torrents"> Options +Indexes AllowOverride None </Directory>
   3.2 Your problem might be another but that for sure must be working, being able to see the files from the browser
   4 How should be the magnet you pass to the WebTorrent class like the demo they post
   4.1
   magnet:?xt=urn:btih:0eb745b6fbe3ba509afc088616e6fe2d1f8d70b2&dn=Folder+Test+With+Spaces&tr=ws%3A%2F%2F__IP__%3A8000&xs=http%3A%2F%2Fyour.web%2Ftorrents%2FFolder%20Test%20With%20Spaces.torrent&ws=http%3A%2F%2Fyour.web%2Ftorrents

4.2 Using for instance this link you can encode and decode:
https://meyerweb.com/eric/tools/dencoder/

4.3 Fully decoded looks like this:
magnet:?xt=urn:btih:0eb745b6fbe3ba509afc088616e6fe2d1f8d70b2&dn=Folder Test With Spaces&tr=ws://__IP__:8000&xs=http://your.web/torrents/Folder Test With Spaces.torrent&ws=http://your.web/torrents

5 As a resume:
&dn= folder name
&tr=ws:// tracker address, if it's https is wss://
&xs= is the torrent file accesible by the web
&ws= the parent folder of the content

Being the unique viewer jumps hundres of errors "many peer connections" etc, which I don't know why... but the webseeding works like this. Hope help you guys @ziro-official @SilentBot1

[ziro-official]

Thanks all, I'm adding the torrent with the .torrent file, I'll provide the sample code and the .torrent file & content file tomorrow.

Thank you.

[ziro-official]

Log from terminal

datnv@lt29-datnv:~/tmp$ npm i webtorrent
npm WARN saveError ENOENT: no such file or directory, open '/home/datnv/tmp/package.json'
npm WARN enoent ENOENT: no such file or directory, open '/home/datnv/tmp/package.json'
npm WARN tmp No description
npm WARN tmp No repository field.
npm WARN tmp No README data
npm WARN tmp No license field.

+ webtorrent@0.105.0
added 5 packages from 5 contributors and audited 1736 packages in 3.246s
found 35 vulnerabilities (3 low, 27 moderate, 5 high)
  run `npm audit fix` to fix them, or `npm audit` for details
datnv@lt29-datnv:~/tmp$ node webtorrent.js 
(node:19760) Warning: N-API is an experimental feature and could change at any time.
(node:19760) [DEP0013] DeprecationWarning: Calling an asynchronous function without callback is deprecated.
Client is downloading:  d0484afd07c8e4f3c09d6e3c370454a6b8df917d
100MB.test
2019-07-22T03:22:12.194Z
2019-07-22T03:22:13.198Z
2019-07-22T03:22:14.199Z
2019-07-22T03:22:15.200Z
2019-07-22T03:22:16.201Z
2019-07-22T03:22:17.201Z
2019-07-22T03:22:18.202Z
2019-07-22T03:22:19.203Z
2019-07-22T03:22:20.203Z
2019-07-22T03:22:21.204Z
2019-07-22T03:22:22.205Z
2019-07-22T03:22:23.206Z
2019-07-22T03:22:24.207Z
2019-07-22T03:22:25.208Z
2019-07-22T03:22:26.208Z
2019-07-22T03:22:27.209Z
2019-07-22T03:22:28.209Z
2019-07-22T03:22:29.209Z
2019-07-22T03:22:30.210Z
^C
datnv@lt29-datnv:~/tmp$ npm uninstall webtorrent
npm WARN saveError ENOENT: no such file or directory, open '/home/datnv/tmp/package.json'
npm WARN enoent ENOENT: no such file or directory, open '/home/datnv/tmp/package.json'
npm WARN tmp No description
npm WARN tmp No repository field.
npm WARN tmp No README data
npm WARN tmp No license field.

removed 5 packages and audited 1298 packages in 2.397s
found 35 vulnerabilities (3 low, 27 moderate, 5 high)
  run `npm audit fix` to fix them, or `npm audit` for details
datnv@lt29-datnv:~/tmp$ npm i webtorrent@0.99.0
npm WARN saveError ENOENT: no such file or directory, open '/home/datnv/tmp/package.json'
npm WARN enoent ENOENT: no such file or directory, open '/home/datnv/tmp/package.json'
npm WARN tmp No description
npm WARN tmp No repository field.
npm WARN tmp No README data
npm WARN tmp No license field.

+ webtorrent@0.99.0
added 23 packages from 16 contributors and audited 1766 packages in 3.541s
found 35 vulnerabilities (3 low, 27 moderate, 5 high)
  run `npm audit fix` to fix them, or `npm audit` for details
datnv@lt29-datnv:~/tmp$ node webtorrent.js 
(node:19841) Warning: N-API is an experimental feature and could change at any time.
(node:19841) [DEP0013] DeprecationWarning: Calling an asynchronous function without callback is deprecated.
Client is downloading:  d0484afd07c8e4f3c09d6e3c370454a6b8df917d
100MB.test
just downloaded: 65536
total downloaded: 844032
download speed: 37449.142857142855
progress: 0.00844032
2019-07-22T03:22:54.637Z
just downloaded: 65536
total downloaded: 909568
download speed: 52428.8
progress: 0.00909568
just downloaded: 65536
total downloaded: 975104
download speed: 71493.81818181818
progress: 0.00975104
2019-07-22T03:22:55.639Z
just downloaded: 65536
total downloaded: 1040640
download speed: 74898.28571428571
progress: 0.0104064
just downloaded: 65536
total downloaded: 1106176
download speed: 93622.85714285714
progress: 0.01106176
just downloaded: 65536
total downloaded: 1171712
download speed: 104857.6
progress: 0.01171712
2019-07-22T03:22:56.640Z
just downloaded: 65536
total downloaded: 1237248
download speed: 101944.88888888889
progress: 0.01237248
just downloaded: 65536
total downloaded: 1302784
download speed: 116508.44444444444
progress: 0.01302784
2019-07-22T03:22:57.641Z
^C
datnv@lt29-datnv:~/tmp$

Source Code

var createTorrent = require('create-torrent')
var fs = require('fs')
var WebTorrent = require('webtorrent')
var client = new WebTorrent()
createTorrent('/home/datnv/Downloads/100MB.test', function (err, tor) {
  if (!err) {
    // `torrent` is a Buffer with the contents of the new .torrent file
    fs.writeFileSync('100MB.test.torrent', tor)

    client.add('100MB.test.torrent', function (torrent) {
      setInterval(() => {
        console.log(new Date())
      }, 1000)

      torrent.addWebSeed('http://66.225.198.198/100MB.test')
      torrent.on('download', function (bytes) {
        console.log('just downloaded: ' + bytes)
        console.log('total downloaded: ' + torrent.downloaded)
        console.log('download speed: ' + torrent.downloadSpeed)
        console.log('progress: ' + torrent.progress)
      })

      console.log('Client is downloading: ', torrent.infoHash)
      torrent.files.forEach(function (file) {
        console.log(file.name)
      })
    })
  }
})

Webseed File:
http://66.225.198.198/100MB.test

Torrent File
please unzip the attached files
100MB.test.torrent.tar.gz

[SilentBot1]

Hi @ziro-official

Thank you for providing us with the example code to help us determine the issue.

After using your test code, it appears that WebSeeds are currently broken in the latest version of WebTorrent (v0.105.0). As described, when attempting to add a WebSeed to a torrent, either via an embedded urlList or torrent.addWebSeed(), no form of HTTP request is made outbound; no head request, no options request, nothing.

I tested this by adding the given torrent file to webtorrent.io which runs on WebTorrent 0.105.0 and then called client.torrents[0].addWebSeed('http://66.225.198.198/100MB.test') and watching for any requests being made within Chrome's network tab. To rule out http/https mismatch and other browser quirks, I tested the library within Node and also was also able to replicate the issue.

After doing some further testing with WebTorrent in Node, WebSeeds stop working in version 0.101.1.

In version 0.101.0 WebSeeds are working correctly and downloading data from the WebSeed but as mentioned in any version above 0.101.0 up to the latest release, 0.105.0, WebSeeds no longer correctly download.

@feross it appears that this specific commit ,8cfab05, between version 0.101.0 and 0.101.1 may have caused the broken functionality of WebSeeds.

[feross]

While i was looking at the sintel.torrent source i could not find any information regarding a xs in the torrent file?
That would result in a magnet link not having any xs when they are generated

There's no way to include xs in .torrent files as far as I'm aware. There's only url-list.

After using your test code, it appears that WebSeeds are currently broken in the latest version of WebTorrent (v0.105.0)

It appears that web seeds work fine in v0.105.0 for sintel.torrent which is deployed on https://webtorrent.io. You can verify that HTTP requests are being sent in the Network tab of the DevTools. Also, you can run client.constructor.VERSION to verify the version is 0.105.0.

Any idea what is causing @ziro-official's torrent to not work?

@feross it appears that this specific commit ,8cfab05, between version 0.101.0 and 0.101.1 may have caused the broken functionality of WebSeeds.

Thanks for tracing the bug to that specific version. Perhaps we made a mistake when converting that file to ES6?

[ziro-official]

@feross I found the root cause of this bug. I change it to wire.interested() and it fixed.
I don't know why @bradleyjkemp added this line but I'm sure it cause our problem. Can i open a pull-request to change it back?