pg_refclone

Clone a PostgreSQL data directory locally using reflink copies.

Usage

python3 pg_refclone.py ORIGINAL_DIR CLONE_DIR

Example:

python3 pg_refclone.py /var/lib/pgsql/18/data /var/lib/pgsql/18/clone

Then start the clone as a standby:

pg_ctl -D /var/lib/pgsql/18/clone -o "-p 5433" start

Requirements

• PostgreSQL 15+ (uses pg_backup_start/pg_backup_stop API)
• Python 3.14+
• psycopg 3
• reflink-capable filesystem (e.g., btrfs, XFS)
• The pg_receivewal binary must be in the same directory as the postgres binary

How it works

1. Connect to the source PostgreSQL instance using the socket from postmaster.pid
2. Create a physical replication slot with a unique random name (pg_refclone_xxxxxx)
3. Start a backup with pg_backup_start()
4. Copy the data directory (using reflinks)
5. Stop the backup with pg_backup_stop()
6. Advance the replication slot to the backup start LSN
7. Stream the required WAL files using pg_receivewal
8. Drop the replication slot

Temporary Replication Slots

We attempted to use a temporary replication slot (created with temporary = TRUE) as an optimization to avoid manual cleanup. The idea was that the slot would be automatically dropped when the session closes.

However, this approach has a fundamental limitation: PostgreSQL's streaming replication protocol is separate from the SQL protocol. To stream WAL, we need to either:

1. Use pg_receivewal (or similar tool) which requires a persistent slot
2. Implement the replication protocol in Python

Currently, psycopg3 does not support physical replication protocol. See: psycopg/psycopg#71

Therefore, we must use a non-temporary slot with a unique name to avoid conflicts when running multiple clones concurrently.

Notes

• The clone directory must not exist or must be empty
• The original and clone directories must be on the same filesystem (for reflink support)
• WAL is not copied from the source; it is streamed separately after the backup
• The clone is started in recovery mode via standby.signal
• Symlinks are not preserved; files are copied directly to avoid data corruption