s3-orchestrator

Project Website · Documentation · Maximizing Free-Tier Storage

Most applications talk to one S3 backend, which ties them to that provider's uptime, pricing, and limits. s3-orchestrator puts a single S3 endpoint in front of any number of S3-compatible backends — OCI Object Storage, Backblaze B2, AWS S3, MinIO, Wasabi, Cloudflare R2, anything that speaks S3 — and presents them as one or more virtual buckets. It tracks where every object lives in its own database, which is what lets it enforce per-backend byte quotas, keep N copies across providers, fail reads over when one goes dark, and take a backend out of the fleet without downtime.

Clients see one endpoint and one namespace. The backends never learn the orchestrator exists — they see ordinary S3 calls, so any provider the AWS SDK can talk to works.

• Stacks providers into one namespace. Cap each backend at a byte limit and writes overflow to the next when it fills, so a 20 GB allocation here and a 10 GB one there become one 30 GB bucket without surprise bills. Monthly API-request, egress and ingress caps work the same way.
• Keeps the copies it promised. Set a replication factor and every object lands on that many distinct backends — placed by the write itself, or by a background replicator. Reads fail over to a surviving copy, a scrubber checks stored bytes against recorded hashes, and an over-replication worker trims the set when a recovered backend brings its copies back.
• Runs at the size you need. Standalone on embedded SQLite with no external dependencies, single-node on PostgreSQL, or many instances with Redis-backed shared counters so quotas hold globally without the nodes coordinating directly.
• Gives operators primitives instead of scripts. Online drain, rebalance, import of a bucket you already have, integrity scrub, a cleanup queue with a dead-letter table, hot-reloadable config, an admin API, a web dashboard, and a terminal object browser.
• Optional at-rest layers. Envelope encryption (AES-256-GCM, with the master key inline, in a file, or in Vault Transit) and chunked zstd compression, both transparent to clients. With both on, compression runs first, because ciphertext does not compress.

Point the orchestrator at the bucket you already have and import its objects into the metadata layer — nothing moves. Add the new provider and raise the replication factor, and the workers copy everything across while traffic keeps flowing. Once the copies are in place, drain the old backend and delete it from the config. No step takes the application down.

If you've gone looking for a tool that does something similar, there don't appear to be many options:

Prerequisites: Go 1.27+, Docker, Make.

git clone https://github.com/afreidah/s3-orchestrator.git
cd s3-orchestrator
make run

Starts three MinIO backends via Docker Compose, embedded SQLite as the metadata store, and the orchestrator on localhost:9000.

aws --endpoint-url http://localhost:9000 s3 cp /etc/hostname s3://photos/test.txt
aws --endpoint-url http://localhost:9000 s3 ls s3://photos/

Default credentials: access key photoskey, secret photossecret. Web dashboard at localhost:9000/ui/ (login admin / admin).

Full credentials and troubleshooting: docs/quickstart.md.

Database: SQLite is embedded — no external dependencies for single-instance use. PostgreSQL 14+ is also an option and is required for multi-instance deployments (database.driver: postgres); the schema migrates on boot.

Generate a config interactively: s3-orchestrator init.

Container images and release checksums are signed with cosign (keyless / Sigstore):

# Container image
cosign verify ghcr.io/afreidah/s3-orchestrator:<version> \
  --certificate-identity-regexp='github\.com/afreidah/s3-orchestrator' \
  --certificate-oidc-issuer='https://token.actions.githubusercontent.com'

# Release checksums
cosign verify-blob checksums.txt --bundle checksums.txt.bundle \
  --certificate-identity-regexp='github\.com/afreidah/s3-orchestrator' \
  --certificate-oidc-issuer='https://token.actions.githubusercontent.com'

              S3 clients (aws cli, rclone, etc.)
                          |
                          v
                    +-----------+
                    | S3 Orch.  |  <-- SigV4 auth, rate limiting, quota routing
                    +-----------+
                     |         |
            +--------+         +------------------+------------------+
            v                  v                  v                  v
       PostgreSQL        OCI Object         Backblaze B2          AWS S3
       (metadata)       Storage (20 GB)       (10 GB)             (5 GB)
                              \                  |                  /
                               '------------ 35 GB total ---------'

Metadata (object locations, quota counters, multipart state, cleanup queue) lives in PostgreSQL or SQLite. Backends only ever see plain S3 calls — no orchestrator-specific protocol, no schema requirements. Any provider that speaks the AWS SDK works.

Deeper details: docs/architecture.md.

Contributions welcome. Start with CONTRIBUTING.md for the build / test / submit workflow, and docs/style-guide.md for the codebase's conventions.

MIT