Skip to content
Data deduplication engine, supporting optional compression and public key encryption.
Rust Shell Makefile Other
Branch: master
Clone or download
dpc Merge pull request #167 from misuzu/passphrase-file
Added RDEDUP_PASSPHRASE_FILE env variable
Latest commit a5ef058 Jan 3, 2020
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
docs/examples Added example documentation for usign rdup. (#68) Jan 26, 2017
lib Update crossbeam, eliminating a redundant crossbeam-utils Dec 31, 2019
scripts Bump libsodium, now bundled with -sys crate Dec 30, 2019
sgdata cargo fmt May 12, 2018
src Handle RDEDUP_PASSPHRASE_FILE reading error with a normal flow Jan 2, 2020
tester
.editorconfig churn: Add auto-formatting for vim Feb 24, 2017
.envrc envrc: use nix Feb 10, 2019
.gitignore Improvements Sep 13, 2017
.travis.yml Rework CI, consolidating {before_,}install and making script fail fast Dec 31, 2019
.vimrc churn: Add auto-formatting for vim Feb 24, 2017
CHANGELOG.md Prepare 3.1.0 Jan 28, 2019
Cargo.lock Bump rpassword version Dec 31, 2019
Cargo.toml Bump rpassword version Dec 31, 2019
HACKING.md Mention existing tests in HACKING.md Oct 20, 2017
Makefile Allow downloading packages, but not updating the lockfile Dec 31, 2019
README.md Update README.md Sep 16, 2019
README.tpl minor improvements / rearrangements Dec 8, 2019
rustfmt.toml cargo fmt May 12, 2018
shell.nix shell.nix: fix openssl, export env vars, refactor Feb 10, 2019

README.md

Travis CI Build Status crates.io Gitter Chat

rdedup

rdedup is a data deduplication engine and a backup software. See current project status and original use case description wiki pages.

rdedup is generally similar to existing software like duplicacy, restic, attic, duplicity, zbackup, etc., with a skew towards asymmetric encryption and synchronization friendly data model. Thanks to Rust and solid architecture, rdedup is also exteremely performant and very reliable (no data-loss bugs ever reported).

rdedup is written in Rust and provides both command line tool and library API (rdedup-lib). The library can be used to embed the core engine into other applications, or building custom frontends and tools.

Features

  • simple but solid cryptography:
    • libsodium based
    • public-key encryption mode (the only tool like that I'm aware of, and primary reason rdedup was created)
  • flat-file synchronization friendly (Dropbox/syncthing, rsync, rclone)
  • immutable data-conflict-free data store
  • cloud backends are WIP
  • incremental, scalable garbage collection
  • variety of supported algorithms:
    • chunking: fastcdc, gear, bup
    • hashing: blake2b, sha256
    • compression: zstd, deflate, xz2, bzip2, none
    • encryption: curve25519, none
    • very easy to add new ones
    • check rdedup init --help output for up-to-date list
  • extreme performance and parallelism - see Rust fearless concurrency in rdedup
  • reliability focus (eg. rdedup is using fsync + rename to avoid data corruption even in case of a hardware crash)
  • built-in time/performance profiler

Strong parts

It's written in Rust. It's a modern language, that is actually really nice to use. Rust makes it easy to have a very robust and fast software.

The author is a nice person, welcomes contributions, and helps users. Or at least he's trying... :)

Shortcomings and missing features:

rdedup currently does not implement own backup/restore functionality (own directory traversal), and because of that it's typically paired with tar or rdup tools. Built-in directory traversal could improve deduplication ratio for workloads with many small, frequently changing files.

Cloud storage integrations are missing. The architecture to support it is mostly implemented, but the actual backends are not.

Installation

If you have cargo installed:

cargo install rdedup

If not, I highly recommend installing rustup (think pip, npm but for Rust)

If you're interested in running rdedup with maximum possible performance, try:

RUSTFLAGS="-C target-cpu=native" cargo install rdedup --vers ...

In case of troubles, check rdedup building issues or report a new one (sorry)!

Usage

See rdedup -h for help.

Rdedup always operates on a repo, that you provide as an argument (eg. --dir <DIR>), or via environment variable (eg. RDEDUP_DIR).

Supported commands:

  • rdedup init - create a new repo.
    • rdedup init --help for repository configuration options.
  • rdedup store <name> - store data from standard input under a given name.
  • rdedup load <name> - load data stored under given name and write it to standard output.
  • rdedup rm <name> - remove the given name.
  • rdedup ls - list all stored names.
  • rdedup gc - remove any no longer reachable data.

In combination with rdup this can be used to store and restore your backup like this:

rdup -x /dev/null "$HOME" | rdedup store home
rdedup load home | rdup-up "$HOME.restored"

rdedup is data agnostic, so formats like tar, cpio and other will work, but to get benefits of deduplication, archive format should not be compressed or encrypted already.

RDEDUP_PASSPHRASE environment variable

If RDEDUP_PASSPHRASE is defined, it will be used instead of interactively asking user for password.

License

rdedup is licensed under: MPL-2.0

You can’t perform that action at this time.