Is 16MiB a hard limit for max pieces size?

[Lyken17]

Hi author,

Thanks for sharing the great tools! I am using your implementation to prototype torrent seeding for large language models in huggingface. Those files (usually >10GB) will lead to many chunks thus make .torrent file itself very large.

I am considering whether we should increase the piece_length in calculate_piece_length(), but noticing there is one comment saying

Proceeding:
1. Start with 256 KIB.
2. While piece count > 2000: double piece length.
3. While piece count < 8:    use half the piece length.

However, enforce these bounds:
- minimum piece length = 16 KiB.
- maximum piece length = 16 MiB.

I checked the specification and documents for torrent protocal but didn't find the source of 16MiB. I wonder is this a hard limit?

[rsnitsch]

This is a soft limit. It only affects the automatic piece size calculation. You can override/specify the piece size yourself with the -p option. Above 16 MiB you will be prompted to confirm the excess of 16 MiB, but you can disable this prompt by adding -f (force option).

For example to create a torrent with 64 MiB: py3createtorrent tests\testdata\bigfile-1gb.dat -p 65536 -f

---

Now why does py3createtorrent prefer to keep the piece size limited to 16 MiB in the first place? The BitTorrent spec itself does not explicitly limit the piece size, but many clients only have limited support for large piece sizes.

Quote:

Warning: users are at their own discretion to create .torrent with >= 64 MiB piece size as not every torrent client supports it.
Larger piece sizes are only available when using libtorrent 2.x. libtorrent 1.x is not efficient with memory usage and in order to avoid user complaints it is limited to 128 MiB.
Also note that, as of this writing, libtorrent 2.0.9 has an internal limitation that only allows loading maximum 256 MiB piece size. And therefore > 256 MiB size options are forbidden for now.

Source: qbittorrent/qBittorrent#19535

The 16 MiB soft limit, however, is from over 10 years ago. Today it should be safe (widely supported) to use 64 MiB pieces. So I will increase the soft limit from 16 MiB to 64 MiB.

[Lyken17]

Wow, apprecaite for such a detailed explaination! Yup, increasing 16MB to 64MB can reduce up to 4x of the torrent file size and definitely will help my project. Thanks!