SHA256-hash every file in a directory tree, saving paths, file sizes and hashes to a SQLite database
shafs can be used to find duplicate files, sort files by size and so forth
shafs requires a valid GNU standard C build system and sqlite3 version > 3.24.0*
On Ubuntu:
sudo apt install sqlite3
sudo apt install libsqlite3-dev
Please check your distribution for the appropriate packages.
*Note: SQLite3 versions < 3.24.0 will not work unless the SQL queries are modified to not use the UPSERT feature. This might generate lots of duplicates in the DB, which you'll need to take into consideration in your queries.
git clone https://github.com/avaxbuildr/shafs.git
cd shafs
./build.sh
To install locally, uncomment last line in build.sh and re-run it or run :
sudo make install
git clone https://github.com/avaxbuildr/shafs.git
cd shafs
mkdir -p m4
autoreconf --install
./configure
make
sudo make install
shafs [-v] <src_dr> <sqlite_file>
shafs -v /usr/lib ~/usrlib_hashed.db
$ shafs ~/ebooks/ ~/ebooks.db
$ sqlite3 ~/ebooks.db
sqlite> select filepath, count(*) as duplicates from shafs group by filehash having duplicates > 0 order by duplicates desc;
sqlite> select * from shafs order by filesize desc;
shafs is released under the terms of the MIT license
See attached LICENSE file for details
Shoutouts: