Quickly detect already witnessed data.
-
Updated
Jul 16, 2017 - Go
Entity resolution (also known as data matching, data linkage, record linkage, and many other terms) is the task of finding entities in a dataset that refer to the same entity across different data sources (e.g., data files, books, websites, and databases). Entity resolution is necessary when joining different data sets based on entities that may or may not share a common identifier (e.g., database key, URI, National identification number), which may be due to differences in record shape, storage location, or curator style or preference.
Quickly detect already witnessed data.
Just some durrdy code to move files around and organize shit...mostly photos and videos.
Golang Deduplication Utility Library
S3 backed FUSE Filesystem written in Go with dedup and encryption.
A tool that deduplicates lines of a textfile with the speed of ram and scales nicely on all cores concurrently.
FLoC, the Flux Capacitor, a set of command line tools to implement backups.
S3 compatible data deduplication and client side encryption program
Content-Addressable File System (used by BitWrk)
New project: https://git.sr.ht/~tsileo/blobfs
Find and obliterate duplicate files, but only the ones you don't care about.
You personal database. Mirror of https://git.sr.ht/~tsileo/blobstash
Fast and cheap partial file hashing provided as a CLI tool and a zero-dependency Go library.
Batch your call. Easily backpressure. Enjoy the performance.
Recursively remove duplicate files in a filesystem.
used to get the dirs/files tree on the disk, including meta, sha1, and record to the sqlite database, then deduplications, make and sync virtual links for dir and files, etc.
Generic simple workflows and concurrency patterns
A simple hash based photo collation and merge program for UNIX systems.
Created by Halbert L. Dunn
Released 1946