Implement private torrents (BEP 27)

[jlouis]

Implement the private BEP, BEP 27. To do this, we must go through:

• The multi-tracker extension
• The DHT
• In the longer run, PEX, if enabled.

[edwardw]

Doing research on this one. I guess the hard part is to figure out how to do the test, ideally automatic one.

[jlouis]

Well, the .torrent file will have a flag that marks the torrent is private. So if you see that, you probably should not start the relevant subsystems. It is mostly used by private trackers which rely on the upload-download rates of peers to lock out those who do not give back. Also, these sites often supply a unique .torrent announce URL per login to track the user further. The private=1 flag is used in that case in order not to spread the information and keep the single tracker only.

To fix it shouldn't be too hard, but it is one of those important things to get a stake in to be a serious client.

[edwardw]

Yes, it's what this BEP about. But I'm also thinking about how to implement automatic tests for it and for etorrent in general, which are fully self-contained and don't rely on any resources outside of the test suite. E.g., I may need to implement a mini tracker. This way, I will feel more confident about my BEP implementation. It also helps if someone wants to change the code in the future.

[edwardw]

BEP 12, the multi-tracker extension, is already in etorrent_tracker_communication, isn't it?

[jlouis]

Yep, BEP12 is in that module. Also, the algorithm used for BEP12 and BEP15 interaction is in the documentation directory (formatted as a BEP, but I never got around to submit it)

[jlouis]

As for the tests, I have been contemplating a ct_ test suite and perhaps test_server. We already have quickcheck mini and eunit for internal tests, but for some real blackbox testing, CT would be really nice to have.

What I tend to do here is to run some exploratory testing on next, but it would be nice to have a simpler test suite you could just run, I agree.

[edwardw]

Nice, both the BEP-12 BEP-15 interaction documentation and the upcoming test suite!

[jlouis]

First stab at making testable code: A branch which can take an alternative configuration at the top-level supervisor, so we can start the etorrent application in a test setting. This means we can get cracking on a black-box test suite soon.

[edwardw]

One quick question, in etorrent_tracker_communication line 271:
etorrent_bcoding:get_value("trackerid", BC, tracker_id_not_given)
But I couldn't find a BEP that mentioned such trackerid thing. Any pointer?

The reason is that in order to conform to BEP 27, I figure etorrent needs to maintain a mapping between a tracker and peers it returns. In http tracker case, if there's a trackerid, it would be very convenient.

[jlouis]

Search for "trackerid" in http://wiki.theory.org/BitTorrentSpecification .. It is an optional value and to be honest, I killed it in a recent commit because we did not use it for anything on subsequent calls.

I think we should reinstate it, but it is not given it will help you given it is optional. Frankly, I have yet to see a tracker returning it though :)

Perhaps you could take the AnnounceUrl given and use it as a marker, eventually with a phash2/1 on it and thread it through so a peer knows where it is coming from. Then when the tracker changes, you can use the peer ETS table to broadcast a messages to all peers on that torrent (and have peers matching the tracker close). You probably also need to send a message to the peer_mgr if it has some peers in its queue matching the tracker identifier.

[jlouis]

Issue #84 has the reinstantiation.

[jlouis]

Merged to master. We can now claim to have BEP 27 support I hope :)