Skip to content

taintedkernel/metasync

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

38 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

************
* MetaSync *
************

Metasync is a file-metadata synchronization tool.  It allows you to automate
propagation of filename and/or directory changes to different remote locations.
It's particularly helpful in an rsync (or-similar)-based workflow in
combination with relatively large amounts of data.

The first use-case was to propagate updates via Lightroom from my photo
collection to several rsync-based backups.  Any sort of semi-frequent file
path updates will benefit from avoiding extra data transfer when syncing
to an external location.

In terms of implementation, metasync keeps an internal database of files
and attemps to determine changes over time through periodic invocation.  If
too many changes happen simultaneously (file contents updated, rename and move)
it may be difficult for metasync to directly detect the correct solution,
in which case it will skip, prompt, or abort - especially if lots
of other similar changes were occuring in the same "snapshot".

Functionality/modes-of-operation:

1. Periodic sync on local source-of-truth filesystem data

   This will detect and interally update any changes to files on disk.
   No changes are made to on-disk bytes, except for the metasync database.

2. Reconcilation with external backup / mirror sync

   When pointed to an external mirror, it will determine what changes
   have not been propagated over and can generate a script which will
   re-create these, or optionally perform the operation itself (if
   direct write access is possible)

3. Bitrot detection

   Performs a forced read-and-recompute hashes for all files.  If any
   data is detected as changed (without corresponding mtime update),
   we can prompt user for action (maybe recompute hash a second time
   to verify not a temporary read error).

   If we have data stored within DB on external mirrors, we can prompt
   user or attempt to recover data.

About

A metadata synchronization tool

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published