DHT does not work (properly)

[Ivshti]

Hello,

I haven't really dug into it but I have tested it several times and am 100% sure - when I start a torrent, it barely connects to 1-2 peers and those connections die quickly. I tried this with two ISP's. Since I remember an older version working perfectly, I went into the commit log and discovered the "using bittorrent-dht instead of vendored one" commit.

So I did

git checkout a4aed6fdb1be5e23639a99dd06e36b24747f28ad

There is a dramatic difference - in this case, everything works as expected.

[mafintosh]

Hmm interesting. Haven't really noticed any performances changes myself. Could you try using the vendored dht impl from the commit you reference in the current master and see if that gives you better performance?

[Ivshti]

Yes. I just went to master, reverted to the vendor DHT (reverting only that commit + fixing the conflict + bringing back dht.js). There are no issues, the performance is as expected.

The problem is not "worse" performance, it's no performance at all. With bittorrent-dht I get a very small amount of peers delivered and I somehow a tiny amount of them end up properly connected ( at best 1-2). I have never managed to start a download with bittorrent-dht.

I am guessing it floods the network with too many UDP requests? No idea.

[mafintosh]

Interesting - Are you using a magnet link or a torrent file? I'm actually pretty sure that bittorrent-dht uses way less udp traffic than my old vendored one. Maybe we can get some input from @feross on this?

[feross]

How are you measuring the amount of udp traffic? It would be interesting to dig deeper into this, but off the top of my head I can't think of an explanation for the difference.

[Ivshti]

I'm using a torrent file. I'll try to measure the amount of UDP traffic later.

[ivantodorovich]

I'm having the same exact issue.

Bad to zero performance on @latest, good performance on @0.4.1 (before a4aed6f).

This happens on some torrents, not all. Maybe the ones the tracker isn't good and DHT is a must...

Take this torrent for instance:
magnet:?xt=urn:btih:6a3c1b2aa35d0d1c2698326b2cadf1f04e5d3bdd&dn=Mixology+S01E07+HDTV+x264-LOL%5Bettv%5D&tr=udp%3A%2F%2Ftracker.openbittorrent.com%3A80&tr=udp%3A%2F%2Ftracker.publicbt.com%3A80&tr=udp%3A%2F%2Ftracker.istole.it%3A6969&tr=udp%3A%2F%2Ftracker.ccc.de%3A80&tr=udp%3A%2F%2Fopen.demonii.com%3A1337

I'm getting ~5 connections on @latest, and ~25 connections on @0.4.1.

[mafintosh]

Hmm. Maybe we should just bring back the old tracker. Someone up for sending a PR?

[ivantodorovich]

Could it be bittorrent-dht is being incorrectly used? I can't find any docs on this module, but looking at https://github.com/feross/webtorrent/blob/037bde2acdf897c8abfc4f0785bd523b3297f1d5/lib/torrent-manager.js#L55

i see they have to manually call dht.listen, and can't find it anywhere in torrent-stream's code.

I'm a bit busy right now, I'll do some tests later on this week.

[mafintosh]

dht.listen shouldn't really matter as its only relevant if you want to be a "server" node in the dht (store data for others).

[ivantodorovich]

You're right.. I don't know what's happening. Peers are being found, but not connecting, apparently.

[mafintosh]

In general the dht returns a lot of invalid peers (peers behind firewalls, etc). Could you do a bench where you test how many peers are returned from the dht using master vs 0.4.1?

[feross]

If we figure out the problem, let's update bittorrent-dht with the fix so others can benefit. I'll debug when I get some time.

[ivantodorovich]

master: 299 peers. Connections: 1/3
@0.4.1: 211 peers. Connections: 12/22

** Connections may or may not be from dht.. could be the tracker

[ivantodorovich]

I've found out that setting the HANDSHAKE_TIMEOUT from 5000 to 30000 improved the results. Getting twice of connections.

I think bittorrent-dht is flooding the connection, as said earlier in this thread.. and peers can't connect until dht finishes, hence almost everyone of them times out.

bittorrent-dht should have a MAX_CONNECTIONS_PER_SECOND or something to avoid flooding..

[ivantodorovich]

Made an ugly implementation of MAX_CONNECTIONS_PER_SECOND, implementing a queue for UDP connections on bittorrent-dht, and the theory got confirmed. Got a good performance, and an increasingly number of connections. (> 20 connections)

I'll cleanup the code and make a PR to bittorrent-dht in a few hours.

[mafintosh]

@ivantodorovich I'm seem to be getting better results using your patch! Your patch is available in torrent-stream 0.6.1 and peerflix 0.5.4

[mafintosh]

@feross Do you have any suggestions in general on how to more scientifically benchmark performance going forward (maybe something that could be useful for webtorrent as well)?

[ivantodorovich]

@mafintosh Once webtorrent/bittorrent-dht#7 gets merged you'll see better results :)

peer connections were timing out because of the network flooding.. Increasing HANDSHAKE_TIMEOUT helps.. but preventing the flood is even better. It's a combination of the two, actually.

[feross]

Just an FYI, webtorrent/bittorrent-dht#7 is merged and published as bittorrent-dht@0.7.0 now.

I agree that going forward we need something more scientific for benchmarking all these modules. I have too much on my plate right now with work and webtorrent, but if no one's tackled it in a month or so, I'll write something for this.

Also, https://github.com/feross/bittorrent-dht has docs now.

[mafintosh]

Looking good @feross. torrent-stream 0.7.1 uses bittorent-dht. 0.7.0. Could you guys (@ivantodorovich, @Ivshti) try out this version?

[Ivshti]

This works properly, good job everyone! I also have to point out that the old bittorrent-dht also crashed my router very often.

Does anyone have an explanation of how the DHT worked properly with mafintosh's dht.js? Did it have a request queue?

[Ivshti]

Also, @mafintosh somehow the version you pushed to npm has a stale "var _ = require('underscore')" and I get a "cannot find module underscore" error unless I install it.

[mafintosh]

@Ivshti could you share the stack so I can see the line? (On my mobile)

[Ivshti]

Okay, sorry, after a fresh install on my workstation there is no such error. I don't know how that happened...

[Ivshti]

Update: while this does work better, it still floods the connection sometimes. Thousands of peers are found but despite the fact I have a fast connection (but sadly a crappy vendor router which I cannot change), torrent-stream cannot establish any peer connections.

@feross it might be a good idea to take a look of how @mafintosh's dht.js implements limits.

[asapach]

@Ivshti, could you please try this with #24? We need more benchmarks.

[feross]

@Ivshti I'll work on this as soon as I get a chance, but I'm traveling for the next week so I won't be able to work on it until I get back. I would accept a PR.

[asapach]

Hey guys, any updates? Is this still actual for the latest version? How about #22 and #24? Should we close this issue?

[feross]

I'm not aware of any outstanding performance issues related to bittorrent-dht.