___,m##__________________(#mw_______________________________________________________________________
_#####B__________________(#####__________######m_______________##L__________________________________
####E_ mmw. __________,emp_"####_________##b__%## __,mmmw___ ww##kwww __.xmmm_______________________
#### _(######m____x######L__####_________##b__(##n_"W' "##b__""##E"""___%' "###_____________________
4####mm_""%#####x####RE"_.e####M_________##b___##__,######L____##L_____.m######_____________________
__%###B____ "######B_____(###R"__________##Q,,,#E__##Q ##Q____##Q,,, _##Q ###.____________________
___ __"______ ####E_______"_ ____________BBBBBDP____B=B"='BT____"BBBB___B==='"BB____________________
______________####__________________________________________________________________________________
______________####b_______ ________________.,,,__________________________...________________________
__x###B_____.######p_____(###m___________###BB##k________________________4##________________________
.####R%___x#####%####m.___W####m________(## ___"_ _.###### __##L####m__########___x##R##m__ ####_###
####__(######R"___%######L_ ####________4##_______ ##O__ ##__##L___##b___###_____##B...##b___ ##E___
####Q_ RRME" ______ _"RWWL__#### ________##p__ mm _##___ ##O_##L___##b___###_____%##"""""____ ##____
_######__________________(#####O_________"######O__"######O__##L####M____'#####___%#####R__ #######_
___BW#B__________________(##M" ______________________________##L____________________________________
_____________________________________________________________PPP____________________________________
Autonomous, resource-predictable real-time ETL & replication engine built in Rust. > Replaces heavy Kafka + Debezium + Flink clusters with a single static binary (< 20 MB footprint).
DataCopter is an ultra-lightweight, high-performance data replication engine designed for engineers who need real-time data sync without infrastructure bloat. Written in pure Rust, DataCopter streams data directly between databases with zero external runtime dependencies, deterministic memory guarantees (OOM-Safe), and crash resiliency out of the box.
Bi-directional real-time replication between PostgreSQL and ClickHouse (PostgreSQL ↔ ClickHouse).
Runs predictably on 12–32 MB RAM and processes up to 60,000+ rows/sec per CPU core.
In-memory dual Write-Ahead Logging (active.wal / shadow.wal) with SIMD CRC32 checksums and atomic watermarks guarantees At-Least-Once data delivery.
Built-in AIMD algorithm automatically adapts batch sizes based on network RTT, shielding analytical databases like ClickHouse from part fragmentation ("Too many parts" error).
Traditional database Change Data Capture (CDC) and ETL solutions (such as Debezium + Apache Kafka + Apache Flink or Kafka Connect) are powerful, but they come with massive operational overhead. Setting up a multi-node JVM ecosystem just to replicate a few tables between PostgreSQL and ClickHouse is often a major engineering overkill.
DataCopter was engineered to solve this exact problem: pure DB-to-DB streaming replication with zero infrastructure overhead.
| Feature / Metric | Traditional CDC Stack (Debezium + Kafka + Flink) | DataCopter |
|---|---|---|
| Runtime Dependencies | Java JVM (JRE 17+), ZooKeeper/KRaft, Kafka Brokers | Zero (Single static binary, < 20 MB) |
| RAM Footprint | 4 GB – 16 GB+ (JVM Heap overhead & GC pauses) | 12 MB – 32 MB (Predictable OOM-Safe bound) |
| Deployment Complexity | Multi-service YAMLs, ZK nodes, connectors, topic management | 1 single executable + 1 declarative pipeline.yaml |
| PG → CH Throughput (1 CPU Core) | ~15,000 – 30,000 rows/sec (Heavy SerDe & multi-hop overhead) | 58,000 – 100,000+ rows/sec (Zero-Copy Rust RowBinary Stream) |
| CH → PG Throughput (1 CPU Core) | ~8,000 – 12,000 rows/sec | 18,000 – 25,000 rows/sec (Transaction-bound max limit) |
| Data Delivery Guarantee | Manual offset commits, complex Flink state checkpoints | At-Least-Once (Dual-WAL + Atomic POSIX Watermarks) |
| Target Load Protection | Fixed batching (risks ClickHouse Too many parts error) | AIMD Autotuner (Dynamic latency-based backpressure) |
| Cold Start / Boot Time | 30 – 90 seconds (JVM warmup & cluster election) | < 15 milliseconds (Instant startup) |
DataCopter pre-allocates bounded MPSC queues and arena row buffers. Memory consumption remains completely flat under extreme throughput spikes.
Row fields are modified directly inside pre-allocated contiguous memory arenas (Row Arena) without allocating heap memory for String or Vector instances during hot execution loops.
Based on measured network write latency (RTT), the autotuner continuously adjusts batch sizes using an Additive Increase / Multiplicative Decrease algorithm. This shields target analytical databases like ClickHouse from part fragmentation ("Too many parts" error).
Ingested rows are immediately written to an active log (active.wal) with hardware-accelerated SIMD CRC32 checksums. Upon batch dispatch, logs rotate to shadow.wal and are erased only after target storage issues a verified ACK.
DataCopter uses an asynchronous, multi-task architecture powered by Tokio. Data flows linearly through ring buffers and write-ahead logs to guarantee zero data loss during power loss or runtime panics.
graph TD
subgraph Source ["Source Subsystem"]
DB_SRC[("PostgreSQL / ClickHouse")]
end
subgraph DataCopter ["DataCopter Core Runtime Engine"]
READER["Reader Task<br/><i>(Async Stream Fetch)</i>"]
WAL_ACT[("active.wal<br/><i>(Group Commit + CRC32)</i>")]
MPSC_Q["Bounded MPSC Queue<br/><i>(Fixed RAM Capacity)</i>"]
RECYCLER["Row Recycler Channel<br/><i>(Zero-Allocation Buffer Reuse)</i>"]
TRANSFORM["In-Place Transform Engine<br/><i>(Masking / Casting / Renaming)</i>"]
WRITER["Writer Task<br/><i>(AIMD Autotuned Batching)</i>"]
WAL_SHD[("shadow.wal<br/><i>(Pending Ack State)</i>")]
STATE_JSON["state.json<br/><i>(Atomic POSIX Watermark Commit)</i>"]
end
subgraph Target ["Sink Subsystem"]
DB_SINK[("ClickHouse / PostgreSQL")]
end
%% Data Flow
DB_SRC -->|"Fetch Rows"| READER
READER -->|"Append Bytes"| WAL_ACT
READER -->|"Push Row Buffer"| MPSC_Q
MPSC_Q -->|"Pop Row Buffer"| TRANSFORM
TRANSFORM -->|"Execute Rules"| WRITER
WRITER -->|"Batch Insert"| DB_SINK
WRITER -.->|"Recycle Empty Row"| RECYCLER
RECYCLER -.->|"Reuse Pre-allocated Row"| READER
%% WAL Lifecycle
WAL_ACT -.->|"Rotate on Batch Write"| WAL_SHD
DB_SINK -.->|"Network ACK"| WRITER
WRITER -.->|"Clear Log"| WAL_SHD
WRITER -.->|"Persist Watermark"| STATE_JSON
You can spin up DataCopter in seconds either as a zero-dependency benchmark or with full database infrastructure via Docker Compose.
Run a high-throughput synthetic chaos test immediately without installing external databases:
# Clone repository
git clone [https://github.com/your-username/datacopter.git](https://github.com/your-username/datacopter.git)
cd datacopter
# Execute with built-in mock pipeline configuration
cargo run --release -- --config ./examples/pipeline.mock.yamlLaunch a full local stack featuring PostgreSQL, ClickHouse, and DataCopter:
# Spin up local database containers and pre-populated schema
docker-compose up -d
# Execute replication pipeline from PostgreSQL to ClickHouse
datacopter --config ./examples/pipeline.postgres-to-clickhouse.yaml --max-memory 512MB# Install pre-compiled static binary (Linux x86_64 / ARM64)
curl -sSL [https://datacopter.dev/install.sh](https://datacopter.dev/install.sh) | sh
# Verify installation
datacopter --helpDataCopter uses declarative YAML files to define replication topographies. Environment variables (${ENV_VAR}) are supported out of the box.
pipeline_id: "pg_to_ch_production_sync" # Unique identifier for state management
state_file: "./data/state.json" # Path to atomic watermark tracking file
# Source database configuration (Reader)
source_connector:
type: "postgres"
connection_string: "${POSTGRES_URL:-postgresql://datacopter:password@127.0.0.1:5432/postgres_db}"
table_name: "users_source"
# Target database configuration (Writer)
sink_connector:
type: "clickhouse"
connection_string: "${CLICKHOUSE_URL:-[http://datacopter:password@127.0.0.1:8123/?database=clickhouse_db](http://datacopter:password@127.0.0.1:8123/?database=clickhouse_db)}"
table_name: "users_analytics"
# Optional buffer overrides (AIMD dynamic tuning active by default)
buffer_settings:
max_batch_size: 25000
# Zero-Copy In-Place Transformation Pipeline
transform_chain:
# Masking: Obfuscates emails via fast FNV-1a non-cryptographic hashing
- operation: "mask"
field: "email"
mask_type: "email"
# Masking: Anonymizes client IP addresses (zeroes out the last subnet octet)
- operation: "mask"
field: "client_ip"
mask_type: "ip"
# Type Casting: Downscales Int64 (BIGINT) to Int8 (SMALLINT)
- operation: "cast"
field: "status_id"
cast_type: "Int8"
# Column Renaming: Maps source field to target table column
- operation: "rename"
field: "status_id"
to: "status_code"| Database Engine | Source (Reader) | Sink (Writer) | Protocol / Strategy | Status |
|---|---|---|---|---|
| PostgreSQL | ✅ | ✅ | Binary Wire Protocol / Transaction Batching | Production Ready (Tier-1) |
| ClickHouse | ✅ | ✅ | HTTP JSONCompactEachRow / RowBinary |
Production Ready (Tier-1) |
| Synthetic / Mock | ✅ | ✅ | In-Memory Chaos Stream & Latency Storm Simulator | Testing Ready |
| MySQL | 🔄 | 🔄 | Binlog CDC / Multi-Row Insert | In Development |
| DuckDB / SQLite | 🔄 | 🔄 | Direct Embedded C-API / File I/O | In Development |
| Apache Kafka | 🔄 | 🔄 | Native Async Streaming Protocol | In Roadmap |
| AWS S3 / MinIO | 🔄 | 🔄 | Streaming Parquet / Compressed CSV | In Roadmap |
DataCopter was benchmarked on standard cloud hardware (4 vCPU, 8 GB RAM) processing 10,000,000 records under simulated network jitter and latency spikes.
Because analytical columnar engines and relational transactional engines process writes differently, throughput varies by replication direction:
- Baseline:
58,400+ rows/secon a single CPU core. - Scalability Note: This baseline is a conservative fraction of DataCopter's capability. Thanks to zero-copy arena buffers and ClickHouse's native
RowBinarystream protocol, throughput scales far higher (100,000+ rows/sec) given sufficient CPU and network bandwidth.
- Baseline:
18,000 – 25,000 rows/sec. - Engine Bottleneck: Naturally limited by relational transactional overhead (PostgreSQL transaction commits, binary parameter binding, and WAL sync). DataCopter maximizes this direction via multi-row parameterized transactions and adaptive connection socket recovery.
- Flatline Memory Footprint (
INV-003): Strict RAM allocation cap maintained between 14.2 MB – 22.8 MB throughout the entire lifecycle. Memory usage remains a flat line under extreme backpressure spikes (OOM-Safe guarantee). - Crash Recovery Latency: Cold recovery (
shadow.wal+active.walevacuation) completes in < 780 ms following an ungracefulkill -9or hard system failure.
DataCopter is open-source software licensed under the GNU Affero General Public License v3.0 (AGPL-3.0).
Contributions from the global open-source community are highly welcome! Check out CONTRIBUTING.md to get started with setting up your development environment and submitting Pull Requests.
If DataCopter reduced your cloud infrastructure bill and replaced heavy Kafka/Flink clusters with a single static binary, consider supporting the developer:
| Coin | Address | Wallet |
|---|---|---|
| BTC | bc1qu4edhpk0wsdzh90aa6f488sr2ylr0rfrn37gt6 | Trust Wallet |
| ETH | 0x5B17758d7Eb8e14119f5Bdf6c2bBD4de786c5d08 | Trust Wallet |
| SOL | HBNsS5XnFFWr7Tye5mGKC7nRUUn1kVCqmEjGtte55G23 | Trust Wallet |
| USDT | 0x5B17758d7Eb8e14119f5Bdf6c2bBD4de786c5d08 | Trust Wallet |
| USDT | 0x5B17758d7Eb8e14119f5Bdf6c2bBD4de786c5d08 | Trust Wallet |
| GRAM | UQCA8hNAxXSNf7M1rWbY4TUXWS6nDdZjNIY6zraubPSo9EYu | Telegram |
(To suppress the donation prompt upon pipeline completion, pass the --no-donation CLI flag or set DATACOPTER_NO_DONATION=1).
Need a dedicated connector for a proprietary database (e.g., Oracle, MS SQL, SAP, custom REST/gRPC APIs), turnkey cluster deployment, or a commercial license without AGPL-3.0 restrictions?
I provide engineering services and custom integration solutions for enterprises:
- 🔌 Bespoke Connector Development: Fast implementation of specialized Source/Sink connectors tailored to your target storage or internal protocols.
- ⚡ On-Premise Tuning & Setup: Performance audits, infrastructure alignment, and high-throughput pipeline configuration.
- 📜 Commercial & Dual Licensing: Flexible licensing options if your organization cannot use AGPL-3.0 open-source code.
📧 Get in Touch: me@chyng.one | Telegram: @chyngalgan