Skip to content

add more chunkers, create separate project for chunkers #9280

@ThomasWaldmann

Description

@ThomasWaldmann
  • evaluate modern chunking algorithms (fastcdc, ultracdc, ...)
  • implement chunking algorithms for usage within borg (chunker written in Cython or from suitable, well-maintained existing external package, tests, interface with existing borg code)
  • evaluate deduplication and throughput performance
  • security: avoid fingerprinting and other attacks on the chunker
  • rip out the borg.chunkers package from borg and create a borgchunk separate project with own tests and documentation

This task does not require much knowledge about the borg code base, but will require evaluating research papers, Cython coding, doing practical experiments, etc. - some math/crypto/security/algorithms background is useful here.

I recently converted our buzhash(32) code from C to Cython, then added a quite similar buzhash64 chunker (also Cython). So all code now is in Cython (some also in Python), no C anymore.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions