Add support for webseeds

[naf419]

Initial attempt.

I tried to follow the psuedo-spec at http://getright.com/seedtorrent.html

(Barely) Tested against archive.org samples.

[coveralls]

Coverage increased (+0.2%) to 18.207% when pulling b27e5e6 on naf419:webseed into c167c5a on rakshasa:master.

[chros73]

Looks interesting! Let me know when you feel it'ts ready to be tested and I'll include it in rtorrent-ps-ch on a branch to be able to test it and I'll post on the original issue.

@rakshasa , this one needs your guidance. :)

@naf419, Thank you for your work!

[naf419]

@chros73, well before you try it, let me tell you what i think it does:

it should:

• throw a dummy tracker on the tracker list for each url in the url-list
• download 1 needed torrent piece at a time from a randomly selected url from the url-list via whatever-protocol-curl-supports (only tested against http)
• remove urls from the url-list the first time it receives an error (cant connect / not found / bad hash)
• stop trying when it has no un-errored urls left
• add to the torrent's and the global download speed indicators
• cleanly finish current piece and stop when torrent is stopped

it wont:

• honor the max download choke speed
• handle threads eloquently. currently spawns 1 thread per torrent-with-webseeds.

given that, if you'd like to test it, have at it.

[chros73]

Thanks for the detailed comment. Do youbplan to modify the functionalty further or this is the last take?

[naf419]

@chros73 no planned functionality changes

[chros73]

I couldn't compile libtorrent :( :

• your patch is fine (luckily without any conflicts)
• but the building env (set up by the build script) is not: libtorrent doesn't find the (headers of) rpath-linked libcurl during compilation
  • the weird thing is that rtorrent does (it also uses the same libcurl)

I couldn't find the cause of the issue "quickly". I don't/won't have time to "play" with it in the near future, sorry.
If I managed to figure out something I'll let you know.
Thanks for your work!

[chros73]

I played with it today and managed to compile it:

• I've modified configure.ac of your patch
• I've also added 1 more line into libtorrent.pc.in

I had to add back setting CPPFLAGS into build script, otherwise the include CFLAGS doesn't get passed to libtool for some reason that I still don't understand why:

• the generated config.status, config.log, libtorrent.pc, etc seems fine and still libtool doesn't use the specified include CFLAGS directive during compilation otherwise

[chros73]

So, I tried out with an archive.org torrent and it seems to be working, but when I tried to create my own (creating with utorrent 2.2.1) it didn't download it. Have you tried it with your own torrent file?

And does it deal with multi/single-file torrents as well, e.g. as in transmission?

it won't honor the max download choke speed

Probably you meant throttle groups, and not choke groups.
It'd be nice if:

• it would add 1 peer / working url to the peerlist

[naf419]

@chros73 i made some torrents myself and you are right about it not working. i only handled when multiple urls were present. and i didn't handle url's not ending with a / for single-torrent files.
fix coming shortly.

[chros73]

Thanks for the update, I also updated my branch with it.

I've attached torrent files to try out with, let's use this URL for testing (we can create lot of type of torrents from it, and they are small files):

• http://suste.besaba.com/Openwrt_Barrier_Breaker
• no peers, just the webseed URL (otherwise we don't now for sure where the data comes from)

A. It properly throws error to the log console if:

• an non-working url is present, but it doesn't remove from the tracker list, e.g. using this:
  • https://www.dropbox.com/sh/u4ob7e8k23icpx8/AADKQSv4m-FKxfCYrut-GVQia/ducks-take-off-2160p-x264.mkv?dl=0
• ducks-take-off-1f-2t-pub.zip

B. It's working with single file torrents:

• no seeds just the webseed URL that doesn't include the actual filename but the trailing slash:
  • http://suste.besaba.com/Openwrt_Barrier_Breaker/TP-LINK-TL-WR842N-ND-V2/
• suste-OpenWRT-BB-0.8.1-tl-wr842n-v2-squashfs-factory.bin.zip

C. I'm not sure about multi file torrents, what URL do we have to use in these cases? I created torrents from this content:

• http://suste.besaba.com/Openwrt_Barrier_Breaker/TP-LINK-TL-WR842N-ND-V2/ (inlcuding the webrevert subdir as well)

• It doesn't does work with this webseed URL:

  • http://suste.besaba.com/Openwrt_Barrier_Breaker/TP-LINK-TL-WR842N-ND-V2
  • TP-LINK-TL-WR842N-ND-V2.zip
  • I thought it should work!

• It does work with this webseed URL:

  • http://suste.besaba.com/Openwrt_Barrier_Breaker/
  • when using all the files:
    • TP-LINK-TL-WR842N-ND-V2-1.zip
  • when only e.g. 2 files were used from the above file list
    • TP-LINK-TL-WR842N-ND-V2-part.zip

Impressive work, @naf419 ! :)

[naf419]

@chros73 according to my reading of the getright spec, url-list entries can either be:
http://srv/folder/filename <-- no trailing /
or
http://srv/folder/ <-- trailing /, so torrent name appended to url

multi-file torrents files follow the same rule, but also add the sub-file name to the end of the url.

your TP-LINK-TL-WR842N-ND-V2.torrent didnt work because it ends in a slash, but already has the torrent name included in the url prefix. you used (my interpretation of) the correct url in the other 2 multi-file torrents by removing the torrent name.

ETA: strangely, the url you wrote above would have worked, but the actual url in the torrent had a trailing /

[chros73]

Errr, that webseed url doesn't have a trailing slash.... Or am I mistaken?

[naf419]

Ya, the one that didn't work for you had a trailing slash. Should have been like this (the url you wrote without the /) TP-LINK-TL-WR842N-ND-V2-corrected.zip

[chros73]

You were right, sorry (I haven't looked in the generated torrent file): utorrent 2.2.1 added the trailing slash. :(
It works indeed fine after editing the file manually. (I also updated my above comments.)

So, it seems all the URL related issues are good now.
What happens if there are other seeders in the swarm? Will webseed be used?

A. I suggest to modify configure.ac:

• PKG_CONFIG_MODULES related lines :
  • curl is required now
• and also add this , makes more checking

B. Although a lot still has to be done if you (we) want this to get merged. :(

May I suggest to go through the web.* and webseed.* files of transmission and "steal" ideas to use in your tracker/controller, e.g. like this?

I think these classes have to be finished first and then the rest of the integration can be dealt with.

Thanks for your work!

[naf419]

What happens if there are other seeders in the swarm? Will webseed be used?

Yes, it gets the next desired chunk from the same pool of desired chunks as the peer requests do, so it should [untested] work fine in simultaneous cooperation with other seeding peers.

RE: merge, i don't imagine it ever will, I just threw it up here for anyone who wanted to patch it in themselves