File indices are miscalculated for v2 files (BEP 0053)

[kovalensky]

Please provide the following information

libtorrent version (or branch):
2.0.8.0
platform/architecture:
win/64
compiler and compiler version:
boost/1.81.0

When using file indices for v2 torrents, libtorrent will fail to set them correspondingly.

Adding f.e. &so=0,5,7 to the magnet link, won't set downloading of those files corresponding to file tree metadata, since there's no files list.
For hybrid and v2 only files file indices could be calculated using file tree list instead of files, since it has no piece alignment files.

Currently I'm in the process of making of the tool to get rid of duplicate files before downloading via magnet links for v2/hybrid .torrent files, to my regret I can't use this feature as thought, maybe there's a workaround?

Tests:

This magnet link will set downloading status only for the first file for attached torrent. 5th and 7th indices won't be set.
Also if magnet metadata will fetch before saving torrent to client, it will download all files, regardless of what we passed in &so parameter.

magnet:?xt=urn:btmh:1220bb994bde0d68f3a9e5e6855ae901874d7511770785b40ef1dbae9256bea0be1d&dn=test&tr=&so=0,5,7

test.torrent.zip

[arvidn]

I recognize that this is a problem. In fact, it's a problem in the libtorrent API as well. The file_storage class uses pad files internally to align files, even for v2-only torrents.

It would be pretty straight forward to fix this for v2-only torrents, since the pad files are implicit and not part of the user interface. But for hybrid torrents, I don't see a way around including pad files in the file list, since they are already exposed to the user.

Do you have any suggestions on what the exact behaviour should be for the various cases?

[kovalensky]

As far as I know, this feature is not widespread, and I personally have never seen or found a tracker using the &so parameter, even if there was, probably none of them would still include hybrid links.

The .pad folder is not displayed in most new clients, so it would not slightly affect anyone if the hybrid and btv2 only magnet links' file indices were calculated using the file tree list representation.

[arvidn]

It would also be simple to fix it for just the magnet link case, to make those indices disregard pad files, unconditionally. It would be a change in behaviour though

[arvidn]

I'm not sure it's a good idea though. It makes it more complicated to predict the behaviour. Perhaps the specification for what &so= mean need to be updated.

[kovalensky]

If by specification you mean additional of ignoring padding files while selecting indices, I agree.

[ValeZAA]

pad folder is not displayed in most new clients

It is displayed in online parsers like Btdig.

[kovalensky]

Hello @arvidn .
After almost a year, are there any news on this one? I wasn't be able to release new version of my tool since then too 😅
Couldn't let go but notice you got participation on other projects, particularly Chia Network, nice 👍🏻👍🏻

[arvidn]

the most reasonable way to address this, as far as I can see, would be to introduce a new index domain, along side the existing file index.

So, in addition to having file_index_t, also have file2_index_t, which always skips pad files. Is that along the lines of what you have in mind as well?

[kovalensky]

Nice approach actually. This does not contradict the existing implementation.

The way I understood is that the second index behaviour can be triggered using an additional parameter or the info-hash sequence order in magnet links?

[arvidn]

magnet links are more complicated. just because you have a v2 info-hash in the magnet link doesn't mean it's not a hybrid torrent. Likewise, just because you have a v1 info-hash in the magnet link doesn't mean it's not a hybrid torrent.

So, you can't change behavior depending on the magnet link shape. Also, I'm not sure anything really needs to be done for magnet links. The specification could be updated, but generally, the magnet link protocol is machine-to-machine. It's OK for it to be a little bit unintuitive to humans.

I'm open to be convinced otherwise though.

[kovalensky]

just because you have a v2 info-hash in the magnet link doesn't mean it's not a hybrid torrent. Likewise, just because you have a v1 info-hash in the magnet link doesn't mean it's not a hybrid torrent.

I was thinking of hybrids while writing the behaviour in previous response and forgot to specify.

Since padding files are generally service files, the specification should be updated someday.
The only rationale here is to provide users with possibility to download torrents without duplicates, i.e., the popularization of v2 protocol torrents.

Or maybe, for now, it's not the best reason.