MediaSync

MediaSync is a fast, local-first pipeline for big photo and video libraries: walk a directory tree, deduplicate with path or content hashing, transcode legacy containers to MP4 (with optional hardware encode on macOS), and export via a staging folder, rclone, or rsync. A terminal UI shows queues, workers, and upload batches; a headless mode fits CI and servers.

Originally built for a WDTV Live Hub workflow, the defaults are now generic so you can point it at any media root.

Why it exists

• Scalable scanning: Persistent per-directory cache (index.json) skips unchanged folders; bounded queues apply backpressure so memory stays predictable.
• Honest dedup: DEDUP_MODE=path for cheap scans, or content for SHA-256 after workers read files (see benchmarks).
• Modern video output: Converts common legacy formats and DVD VIDEO_TS layouts to MP4; preserves mtimes where possible.
• Efficient cloud upload: Batched rclone runs with configurable parallelism cut per-file overhead.

Requirements

• Rust 1.70+
• ffmpeg and ffprobe on PATH
• rclone (only if UPLOAD_STRATEGY=rclone)
• rsync (only if UPLOAD_STRATEGY=rsync)

Install

git clone https://github.com/ashesh8500/mediasync.git
cd mediasync
cargo build --release
# binary: target/release/mediasync

Or from a checkout:

cargo install --path .

Usage

# Scan current directory; TUI (default state dir ~/.mediasync)
mediasync

# One-shot pass, logs only (good for automation)
mediasync --headless true --run-once true

# Custom source and cloud remote (rclone remote name or `remote:path`)
mediasync --src-root /data/media --upload-strategy rclone --gphotos-remote myremote:albums

Environment variables

Variable	Default	Role
SRC_ROOT	.	Root to scan
STATE_DIR	~/.mediasync	Cache, upload history, conversion output, spool
PHOTOS_WATCH	~/PhotosStaging	Desktop strategy destination
UPLOAD_STRATEGY	desktop	desktop | rclone | rsync
GPHOTOS_REMOTE	gphotos	rclone destination (remote or name:path)
RSYNC_DEST_ROOT	—	Required for rsync strategy
MAX_WORKERS	4	Photo + direct-video workers
VIDEO_WORKERS	2	Transcode workers
DEDUP_MODE	content	content (SHA-256) or path
RCLONE_BATCH_SIZE	50	Files per rclone batch
HEADLESS	false	Disable TUI
RUN_ONCE	false	Exit after one scan + drain
RETRY_FAILURES	false	Retry recent failures (ignores RETRY_AFTER_SEC backoff)

Full CLI help:

mediasync --help

Benchmarks

Streaming SHA-256 (the hot path for DEDUP_MODE=content):

cargo bench -p mediasync --bench hashing

Open target/criterion/report/index.html for Criterion’s HTML report.

Migrating from older wdtv-sync / ~/.wdtv_sync

1. Install the mediasync binary (this crate).
2. Either copy state: cp -R ~/.wdtv_sync ~/.mediasync
   or set STATE_DIR=$HOME/.wdtv_sync to keep the old location.
3. Set SRC_ROOT if you relied on the old default (/Volumes/WDTVLiveHub).

Historical docs live under archive/.

Project layout

Path	Purpose
src/main.rs	CLI, TUI, scanner, workers, rclone batch uploader
src/lib.rs	Library surface (tests, benchmarks)
src/hash.rs	SHA-256 helpers
benches/hashing.rs	Criterion benchmarks

License

MIT — see LICENSE.

Contributing

See CONTRIBUTING.md.