Skip to content

v0.1.0

Choose a tag to compare

View all tags
@github-actions github-actions released this 02 Aug 04:48
· 72 commits to main since this release
v0.1.0
6c40c5b

pgcopy Release Notes

v0.1.0 - Initial Release 🎉

Released: August 1, 2025

We're excited to announce the initial release of pgcopy, a high-performance CLI tool for efficiently copying data between PostgreSQL databases with identical schemas. This tool is specifically designed for data-only migrations and provides parallel processing capabilities for optimal performance.

🚀 What is pgcopy?

pgcopy addresses the common challenges faced when copying data between PostgreSQL databases, particularly in scenarios where traditional tools like pg_dump/pg_restore are too slow or don't work due to permission constraints. It's perfect for development environment data refreshes, database migrations between hosts, and cross-cloud data transfers.

✨ Key Features

Core Functionality

  • High-Performance Data Copying: Parallel table copying with configurable workers for optimal performance
  • Intelligent Foreign Key Management: Automatically detects and manages foreign key constraints, including complex circular dependencies
  • Batch Processing: Configurable batch sizes for optimal memory usage and performance
  • Transaction Safety: Uses transactions to ensure data consistency throughout the copy process

User Experience

  • Real-Time Progress Tracking: Visual progress bar with detailed logging (progress bar stays fixed at top while logs scroll below)
  • Flexible Configuration: Support for connection strings, config files, and command-line options
  • Table Filtering: Include/exclude specific tables from the copy operation using patterns
  • Dry Run Mode: Preview what will be copied without actually copying data
  • Resume Capability: Resume interrupted copy operations from where they left off

Performance & Reliability

  • Parallel Processing: Multiple tables copied simultaneously for significantly faster operations
  • Connection Pooling: Optimized database connections for better resource utilization
  • Memory Efficient: Smart batching prevents memory issues with large datasets
  • Error Recovery: Built-in error handling with automatic constraint restoration

🎯 Use Cases

pgcopy is specifically designed for casual data synchronization scenarios:

  • Development Environment Data Refresh: Quickly sync production data to staging/dev environments
  • Database Migration Between Hosts: Move database data to new servers (data-only migrations)
  • Subset Data Synchronization: Sync only specific tables between environments
  • Cross-Cloud Data Transfer: Transfer data between different cloud providers or regions

📦 Installation

From Source

git clone https://github.com/koltyakov/pgcopy.git
cd pgcopy
make build

Install to System

make install

🚀 Quick Start

Basic Usage

pgcopy copy \
  --source "postgres://user:password@source-host:5432/source_db" \
  --dest "postgres://user:password@dest-host:5432/dest_db"

High-Performance Copy

pgcopy copy \
  --source "postgres://user:pass@source:5432/db" \
  --dest "postgres://user:pass@dest:5432/db" \
  --parallel 8 \
  --batch-size 5000

Selective Copy

pgcopy copy \
  --source "postgres://user:pass@source:5432/db" \
  --dest "postgres://user:pass@dest:5432/db" \
  --include-tables "users,orders,products,inventory"

🛠️ Available Commands

  • pgcopy copy: Copy data between PostgreSQL databases
  • pgcopy list: List tables in a database
  • pgcopy version: Show version information

🔧 Configuration Options

  • Connection Methods: Direct connection strings or configuration files
  • Performance Tuning: Configurable parallel workers and batch sizes
  • Table Filtering: Include/exclude patterns for selective copying
  • Logging: Comprehensive logging with progress tracking
  • Foreign Key Handling: Automatic constraint management

📊 Performance Advantages

Scenario pg_dump/pg_restore pgcopy Advantage
Large tables (>1M rows) Serial processing Parallel batching 3-5x faster
Many small tables Overhead per table Parallel workers 2-4x faster
Complex FK constraints Manual intervention Automatic handling Much easier
Cloud/managed databases Permission issues Works without superuser Actually works

🎯 What pgcopy is NOT

  • Not for online replication: This is not a real-time replication solution
  • Not Change Data Capture (CDC): It doesn't track or sync incremental changes
  • Not for production replication: Use PostgreSQL's built-in logical/physical replication for production
  • Not a schema migration tool: Both source and destination databases must have identical schemas

🔮 Future Roadmap

Planned enhancements for future releases:

  • Enhanced resume capability with checkpoint-based functionality
  • Automatic schema validation and compatibility checking
  • Adaptive batching with dynamic batch size adjustment
  • Advanced connection multiplexing

🤝 Contributing

We welcome contributions! Please see our contributing guidelines in the repository for more information on how to get involved.

📄 License

This project is licensed under the MIT License. See the LICENSE file for details.

Download: GitHub Releases

Documentation: README.md

Issues & Support: GitHub Issues

Assets 9
Loading