Use FastCDC to split chunk

[yuanjingsong]

I tried to use FastCDC algorithm as a split algorithm, and I find it can speed up 20%-30% than the original method when it computes the large files.
In short, FastCDC is a fast chunking algorithm which has a better performance than traditional CDC algorithm. So I tried to use it to speed up chunking phase.
The paper about FastCDC is here

[yuanjingsong]

It's normal if it cant pass the CI test, because the different split algorithm changes the chunking result.

[fd0]

Thanks for your contribution! I'm sorry, this is not something we can merge. There are several programs (including restic, for which I've built this package in 2014, two years before the FastCDC paper) which depend on deterministic output from this library. Changing the chunking algorithm is not possible.

I suggest you publish a chunker package yourself. Please don't use a fork, create a new repo for that . Feel free to use this package as a base, but then please remove every mention of the original chunking method ("Rabin CDC") from all source files, so users can clearly distinguish your new package from this one.

I'm closing this issue for now, please feel free to add further comments!