🔍 File Comparison and Copy Tools

This repository contains two complementary tools for comparing and copying files between directories:

1. 🚀 A Go-based file comparison tool that identifies missing files
2. 📦 A Python-based file copy tool that transfers the identified missing files

🌟 New to macOS setup? Check out our guide for simple installation instructions!

⚙️ Prerequisites

For the Go Tool 🚀

Installing Go with Homebrew

brew install go

• Go 1.22 or later (installed via Homebrew above)
• Required Go packages (automatically installed via go.mod):
  • github.com/schollz/progressbar/v3
  • gopkg.in/ini.v1

For the Python Tool 🐍

• Python 3.6 or later
• Required Python packages:
  • tqdm
  • configparser

🔧 Installation

1. Clone the repository:

git clone git@github.com:NamiLinkLabs/FileCompare.git
cd FIleCompare

2. Install Go dependencies:

go mod download

3. Install Python dependencies:

pip install tqdm configparser

⚡ Configuration

Create a config.ini file in the root directory with the following structure:

[Directories]
source_dir = /path/to/source/directory
target_dir = /path/to/target/directory

[FileTypes]
excluded_extensions = .tmp, .temp, .bak

[Hashing]
large_file_threshold = 104857600
partial_hash_size = 1048576

Note - you must define full (absolute) path! Relative paths are not supported.

📚 Usage

1. Compare Directories 🔍

You can compare directories using either the Go tool directly or the Python wrapper:

Using Go directly:

go run main.go

Using Python wrapper:

from compare import CompareTools

# Initialize the comparison tools
compare = CompareTools()

# Run the comparison
missing_files, total_missing = compare.compare_directories()

# Get cache statistics
cache_stats = compare.get_cache_stats()

You can also run the wrapper directly:

python compare.py

This will:

• Scan the source and target directories
• Create a missing_files.csv containing paths of files present in source but missing in target
• Generate source_dups.csv and target_dups.csv listing duplicate files in each directory
• Generate cache files (source_cache.json and target_cache.json) to speed up future comparisons

2. Copy Missing Files (Python Tool) 📂

After running the comparison, use the Python tool to copy the missing files:

python copyfiles.py

This will:

• Read the missing_files.csv generated by the Go tool
• Create a missed_files directory in the target location
• Copy all missing files while preserving their directory structure
• Show a progress bar during the copy process

⚡ Performance Features

Go Comparison Tool 🚀

• Concurrent file hashing using worker pools
• Smart hashing for large files (only hashes beginning and end)
• File hash caching to speed up subsequent runs
• Duplicate file detection in both source and target directories
• Progress bar for visual feedback
• Handles permission errors gracefully

Python Copy Tool 🐍

• Progress bar for visual feedback
• Preserves file metadata during copy
• Creates necessary directory structure automatically
• Error handling for failed copies

📁 Output Files

• 📄 missing_files.csv: List of files missing from the target directory
• 📄 source_dups.csv: List of duplicate files in the source directory
• 📄 target_dups.csv: List of duplicate files in the target directory
• 💾 source_cache.json: Cache of file hashes from the source directory
• 💾 target_cache.json: Cache of file hashes from the target directory

⚠️ Error Handling

Both tools include error handling for common scenarios:

• Permission denied errors
• Missing directories
• Invalid file paths
• I/O errors during copying

👥 Contributing

Feel free to submit issues and enhancement requests! Pull requests are welcome! 🎉