pg_fast_data_transfer 🚀

pg_fast_data_transfer is a high-performance PostgreSQL data transfer tool written in Go. It is designed to efficiently move data between databases—even across different servers—handling large datasets with ease using streaming COPY protocols and batch processing.

Key features include:

• ⚡ High Performance: Uses PostgreSQL COPY protocol for maximum throughput.
• 🔄 Streaming: Low memory footprint by streaming data directly between connections.
• 📦 Batch Processing: Configurable batch sizes and error handling.
• 🛠️ Flexible Configuration: Configure via .env file or CLI arguments.
• 🛡️ Type Safe: Supports all PostgreSQL data types (JSONB, Arrays, Vectors, etc.).

---

📋 Table of Contents

• Prerequisites
• Installation
• Configuration
• Usage
• Project Structure

🛠 Prerequisites

• Go 1.20 or higher
• PostgreSQL (Source and Destination databases)

📦 Installation

1. Clone the repository:

   git clone https://github.com/username/pg_fast_data_transfer.git
   cd pg_fast_data_transfer

2. Build the binary:

   go build -o pg_fast_data_transfer.exe

⚙ Configuration

The application uses a .env file for configuration. Copy the example file to get started:

cp .env.example .env

Edit the .env file with your database credentials and transfer settings:

# Source Database
SOURCE_DB_HOST=localhost
SOURCE_DB_PORT=5432
SOURCE_DB_USER=postgres
SOURCE_DB_PASSWORD=secret
SOURCE_DB_NAME=source_db
SOURCE_DB_SCHEMA=public

# Destination Database
DEST_DB_HOST=localhost
DEST_DB_PORT=5432
DEST_DB_USER=postgres
DEST_DB_PASSWORD=secret
DEST_DB_NAME=dest_db
DEST_DB_SCHEMA=public

# Tables to Transfer (comma-separated lists must match order)
SOURCE_TABLES=users,orders
DEST_TABLES=users_backup,orders_archive

# Performance Tuning
BATCH_SIZE=10000
TRUNCATE_BEFORE_TRANSFER=false

🚀 Usage

Basic Usage

Run the tool using the configuration from your .env file:

./pg_fast_data_transfer.exe

CLI Arguments

You can override .env settings or run ad-hoc transfers using command-line flags:

Flag	Description	Example
--source-table	Specific source table (schema.table)	--source-table=public.users
--dest-table	Specific destination table	--source-table=backup.users
--truncate	Truncate destination before transfer	--truncate
--help	Show help message	--help

Example: Single Table Transfer

./pg_fast_data_transfer.exe --source-table=users --dest-table=users_backup --truncate

📂 Project Structure

├── config/         # Configuration loading and parsing
├── database/       # Database connection logic
├── transfer/       # Core data transfer and streaming logic
├── .env            # Environment configuration
└── main.go         # Entry point and CLI handling