Custom Bittorrent extensions on nodejs stopped working after v0.81.0

[chr15m]

Test code and instructions for replicating this bug can be found in this repository:

https://github.com/chr15m/webtorrent-extension-bug

This is the script:

https://github.com/chr15m/webtorrent-extension-bug/blob/master/wt.js

What version of WebTorrent?

webtorrent@0.81.0 & webtorrent@0.98.19 under node, not browser.

webtorrent@0.81.0 was the last version where the test extension would work as expected.

What operating system and Node.js version?

node: v8.7.0

Machine 1 (local):
Linux HOSTNAME 3.19.0-80-generic #88~14.04.1-Ubuntu SMP Fri Jan 13 14:54:07 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Machine 2 (remote):
Linux HOSTNAME 2.6.32-5-686-bigmem #1 SMP Tue May 13 16:55:48 UTC 2014 i686 GNU/Linux

What did you expect to happen?

The correct behaviour occurs on on v0.81.0 - the two instances of the extension protocol in wt.js find each other and begin exchanging messages:

$ ./wt.js 
peerId 2d5757303038312d623732633666396333353762
ready {}
torrent 8241104626199aa2b84911e238e10df9484d156c
wire 71.19.157.237:42237 2d5757303038312d353534643065323537386636
t.onHandshake 8241104626199aa2b84911e238e10df9484d156c 2d5757303038312d353534643065323537386636
t.onHandshake extensions { dht: true, extended: true }
t.onExtendedHandshake { m: { sw_testext: 3, ut_metadata: 1, ut_pex: 2 },
  metadata_size: 100,
  test: <Buffer 48 65 6c 6c 6f 2c 20 57 6f 72 6c 64 21> }
Sending payload: 0.1042553525838994
t.onMessage 18:0.8657093983463904
t.onMessage 19:0.06903898600173841
Sending payload: 0.9610726478189229
t.onMessage 18:0.2457831273647444
Client is seeding: 8241104626199aa2b84911e238e10df9484d156c
Sending payload: 0.7495241725092892
t.onMessage 18:0.8740728447699799

What actually happened?

Under the latest version v0.98.19 the two instances do not find each other and their wires do not connect:

$ ./wt.js 
peerId 2d5757303039382d58674354746b7a7a53416d7a
ready {}
torrent d681d6f7da8511270a04bd00e87538d13cac3745
Client is seeding: d681d6f7da8511270a04bd00e87538d13cac3745

[gverni]

I tried to debug this issue and i suspect that the bug may actually be with the previous version of webtorrent (0.81.0) and not with the new one.

With the new version (0.98.19) if a torrent is "done" (as it is the case in the test application, since it's creating a torrent calling seed()), the peer is not added (unless it's manually added calling addPeer() on the torrent object). The check is done in the onPeer function:

webtorrent/lib/torrent.js

Line 355 in a7f8f71

if (typeof peer === 'string' && self.done) return

If this line is commented, then the message are shown as before.

As I mentioned, I suspect that this is not actually a bug, but I'd ask the assistance of @DiegoRBaquero to confirm my analysis (i've just started looking into webtorrent code yesterday)

Hope it helps! Let me know if you need more details on my analysis.
Giuseppe

[chr15m]

@gverni very good, thanks very much for taking the time to figure out exactly which bit of code is responsible! I can confirm that commenting out that line fixes the issue and the correct behaviour continues.

I believe this is a bug in webtorrent because BEP 0010 which specifies this feature does not say anything about ignoring extended message connections after the torrent is done. In fact it says:

Subsequent handshake messages can be used to enable/disable extensions without restarting the connection.

It would be good to see how other implementations handle this situation (torrent "done" but extension messages continue) but on reading the spec this seems like a bug to me.

Perhaps an intermediate fix would be for extension protocol handlers to set a flag saying that they would like to continue handling connections after the torrent has finished downloading?

[chr15m]

@DiegoRBaquero after doing some testing and thinking about this some more I think that @gverni is correct that this is not a bug. If a node is seeding some content we would not expect it to make outgoing connections to other peers.

I managed to solve my particular use-case by using webtorrent.add(hash) instead of using webtorrent.seed(). Using a hash that does not correspond to a real torrent means the node reaches out the swarm and is able to exchange extension messages as the torrent is never in a "done" state.