Use Rabin splitter (data sensitive) by default instead of SizeSplitter (data insensitive)

[sideeffffect]

Why not use the Rabin splitter in IPFS instead of the SizeSplitter by default?
The rolling hash based data sensitive Rabin splitter has a huge obvious advantage: it creates shift-resistant chunks and thus improves data deduplication and sharing.

For example casync is doing it too:

• http://0pointer.net/blog/casync-a-tool-for-distributing-file-system-images.html
• https://github.com/systemd/casync
• https://github.com/folbricht/desync (alternative Go implementation)

[Stebalien]

Primarily, to avoid changing file hashes too often. Our plan is to switch to rabin (or similar), CIDv1, raw leaves, UnixFSv2 (https://github.com/ipfs/unixfs-v2) etc. all in one go.

[sideeffffect]

oh, interesting! any idea when that could happen? is it in weeks, months or years?

[Stebalien]

At this point, hopefully months. We're pushing hard for UnixFSv2 at the moment as it will help us support arbitrary file metadata (important for package manager integration).

However, we could always use suggestions on better chunking algorithms. We picked rabbin out of a hat but there may be better algorithms.

[sideeffffect]

FWIW casync (mentioned above) uses Buzhash

[MarkusTeufelberger]

bup (a backup utility) splits with "an algorithm similar to rsync": https://github.com/bup/bup/blob/master/Documentation/bup-split.md (implementation: https://github.com/bup/bup/blob/master/lib/bup/bupsplit.c)

restic (another backup utility) uses Rabin with a min and max size: https://github.com/restic/restic/blob/master/doc/design.rst#backups-and-deduplication

[sideeffffect]

Hi, has the rolling hash chunker debate moved forward in the recent months?

[ribasushi]

@sideeffffect on my end work has been postponed until Q1-ish of next year (the work that @aidanhs linked right above your comment)
A number of insights were already gathered, including a definitive proof that most current single-pass algorithms (like Rabin) do not perform sufficiently well at the places we are trying to use them at.
More in couple months, deity willing ;)

[sideeffffect]

Has the situation about the chinking strategy changed?

[hacdias]

This repository is no longer maintained and has been copied over to Boxo. In an effort to avoid noise and crippling in the Boxo repo from the weight of issues of the past, we are closing most issues and PRs in this repo. Please feel free to open a new issue in Boxo (and reference this issue) if resolving this issue is still critical for unblocking or improving your usecase.

You can learn more in the FAQs for the Boxo repo copying/consolidation effort.

[sideeffffect]

Reopened in boxo: ipfs/boxo#355