s3-sync

Bidirectional synchronisation between two S3-compatible buckets (AWS S3, MinIO, LocalStack).

Features

• 2-way sync based on object key, timestamp, and ETag/SHA256
• Persistent SQLite state to detect new, changed, and deleted objects
• Conflict detection with configurable resolution strategies
• Dry-run by default — no writes without --execute
• JSON configuration, per-pair settings
• Optional JSON report output

Setup

python -m venv .venv
.venv\Scripts\pip install -r requirements.txt

Credentials

Credentials are not stored in config.json. Use one of:

AWS profiles (recommended for multiple MinIO instances with different credentials):

C:\Users\<you>\.aws\credentials:

[minio-wsl]
aws_access_key_id = your-access-key
aws_secret_access_key = your-secret-key

[minio-linux]
aws_access_key_id = your-access-key
aws_secret_access_key = your-secret-key

Environment variables (simpler, same credentials for both sides):

$env:AWS_ACCESS_KEY_ID = "your-access-key"
$env:AWS_SECRET_ACCESS_KEY = "your-secret-key"

Then set "profile": null in config.json.

Configuration

Edit config.json:

{
  "database": { "path": "data/sync_state.sqlite" },
  "logging":  { "level": "INFO", "file": "logs/s3-sync.log" },
  "defaults": {
    "dry_run": true,
    "delete_mode": "ignore",
    "conflict_strategy": "keep_newer",
    "hash_mode": "etag_then_sha256_on_conflict"
  },
  "sync_pairs": [
    {
      "name": "main-sync",
      "bucket_a": {
        "name": "uploads",
        "prefix": "images/",
        "endpoint_url": "http://wsl:9000",
        "profile": "minio-wsl",
        "region": "us-east-1"
      },
      "bucket_b": {
        "name": "uploads",
        "prefix": "images/",
        "endpoint_url": "http://linux:9000",
        "profile": "minio-linux",
        "region": "us-east-1"
      },
      "sync": {
        "dry_run": true,
        "delete_mode": "ignore",
        "conflict_strategy": "keep_newer",
        "hash_mode": "etag_then_sha256_on_conflict",
        "max_sha256_size_mb": 100
      }
    }
  ]
}

Options

Field	Values	Default	Description
delete_mode	ignore, propagate	ignore	ignore: deleted objects are restored from the other side. propagate: deletion is mirrored.
conflict_strategy	keep_newer, prefer_a, prefer_b, manual	keep_newer	How to resolve objects changed independently on both sides.
hash_mode	etag, etag_then_sha256_on_conflict	etag_then_sha256_on_conflict	ETag is fast; SHA256 is calculated only when ETags differ.
max_sha256_size_mb	integer	100	Objects larger than this are compared by ETag only.

Usage

# Dry-run all pairs (no changes made)
.venv\Scripts\python sync.py --config config.json

# Dry-run a specific pair
.venv\Scripts\python sync.py --config config.json --pair main-sync

# Execute sync
.venv\Scripts\python sync.py --config config.json --execute

# Execute and write JSON report
.venv\Scripts\python sync.py --config config.json --execute --report report.json

# Verbose logging
.venv\Scripts\python sync.py --config config.json --verbose

Sync Logic

Situation	Action
Object only in A (new)	Copy A → B
Object only in B (new)	Copy B → A
Object in both, identical	No-op
Object in both, only A changed	Copy A → B
Object in both, only B changed	Copy B → A
Object in both, both changed	Conflict → resolved per conflict_strategy
Object deleted on one side	Depends on delete_mode

First run: No prior state exists, so all objects are treated as "new" and copied to whichever side is missing them.

Project Structure

s3_sync/
├── sync.py           # CLI entry point
├── config.json       # Configuration
├── requirements.txt
├── data/             # SQLite database (auto-created)
├── logs/             # Log files (auto-created)
└── app/
    ├── config.py     # Config loading and validation
    ├── db.py         # SQLite layer (3 tables)
    ├── models.py     # Dataclasses
    ├── s3_client.py  # boto3 wrapper
    ├── engine.py     # Core sync logic
    └── resolver.py   # Conflict resolution strategies

SQLite Database

Three tables in data/sync_state.sqlite:

• object_state — last known state of every object on both sides
• sync_runs — one row per sync execution with counters and status
• sync_actions — every planned/executed action with reason and result

Browse with any SQLite client (e.g. DB Browser for SQLite).