Could the metalink-hasher create Torrent files in the same step?

[poeml]

                                                                                       [          ]

Issue migrated (2015-06-05) from old issue tracker http://mirrorbrain.org/issues/issue37

Title    Could the metalink-hasher create Torrent files in the same step?
 Priority   wish                     Status           resolved
Superseder                         Nosy List          poeml
Assigned To poeml                   Keywords

msg119 (view) Author: poeml Date: 2010-02-03.22:45:30

Vittorio suggested that the metalink-hasher might create torrent files along the way.

The metalink-hasher is a Python program which walks the file tree and looks what needs
to be done; it calls the "metalink" binary which is a standalone binary.

Darix suggests to look at make_torrent example from libtorrent-rasterbar and either
call that or even link libtorrent-rasterbar into metalink. That way you only need to
read each block once for hashing.

http://www.rasterbar.com/products/libtorrent/
http://www.rasterbar.com/products/libtorrent/client_test.html

Maybe metalink could be extended easily to do the same. Or, maybe the output of
"metalink" could be used - after all, it outputs a series of hashes. However, maybe the
blocks would not fit?

msg120 (view) Author: poeml Date: 2010-02-03.22:48:44

Darix suggests to talk to hydri in #libtorrent.

msg121 (view) Author: poeml Date: 2010-02-04.11:43:28

Darix suggests as first approach to hack make_torrent calls into the python
thingie (metalink-hasher).

See http://www.rasterbar.com/products/libtorrent/examples.html

msg133 (view) Author: poeml Date: 2010-02-24.23:55:12

Looking further, this is all straightforward - metalink-hasher could provide the
hashes in a form digestible by Apache, and they would contain all that's needed.
Apache could easily generated the torrent live.

However the question arises whether it might be better to have real .torrent files
on disk (and maybe even next to the original files), simply for two reasons:

• they can be mirrored then
• to run a seeder (I'm not sure about the exact needs of seeders, but I suppose
  they benefit from already prepared torrent files)

For practical reasons, the latter might be more important than the former.

msg134 (view) Author: poeml Date: 2010-02-25.00:02:38

Links about generation of torrents:

http://en.wikipedia.org/wiki/BitTorrent_(protocol)#Creating_and_publishing_torrent
s
http://www.bittorrent.org/beps/bep_0003.html
http://wiki.theory.org/BitTorrentSpecification
http://en.wikipedia.org/wiki/Bencode
http://pypi.python.org/pypi/BitTorrent-bencode/5.0.8

msg138 (view) Author: poeml Date: 2010-03-08.20:48:00

This requires a hash cache redesign -- tracked in issue 40.

msg169 (view) Author: poeml Date: 2010-03-31.19:29:03

Hash cache redesign largely finished, and in testing now.

Torrent support working already in trunk. Now figuring out things like updating
the tracker servers, and exact bittorrent format that is suitable for as many
clients as possible.

msg173 (view) Author: poeml Date: 2010-03-31.20:04:21

another interesting link, regarding hashes to be included:
http://wiki.depthstrike.com/index.php/P2P:Protocol:Specifications:Optional_Hashes

msg219 (view) Author: poeml Date: 2010-09-05.23:44:55

This issue can be closed as resolved. Way cool!

History
         Date         User  Action             Args
2010-09-05 23:45:03 poeml set    assignedto: poeml
2010-09-05 23:44:56 poeml set    status: testing -> resolved
                                   messages: + msg219
2010-03-31 20:04:21 poeml set    messages: + msg173
2010-03-31 19:29:05 poeml set    status: deferred -> testing
                                   messages: + msg169
2010-03-08 21:20:32 poeml set    status: chatting -> deferred
2010-03-08 20:48:00 poeml set    messages: + msg138
2010-02-25 00:02:38 poeml set    messages: + msg134
2010-02-24 23:55:12 poeml set    messages: + msg133
2010-02-04 11:43:28 poeml set    messages: + msg121
2010-02-03 22:48:44 poeml set    messages: + msg120
2010-02-03 22:45:30 poeml create

(end of migrated issue)