A high-performance remote file transfer utility optimized for efficient bulk transfers over SSH. Copyem uses intelligent file scheduling, parallel transfers, and adaptive buffering to maximize throughput while minimizing transfer time.
- Smart File Scheduling: Optimizes file order to minimize latency and maximize throughput
- Parallel Transfers: Support for multiple concurrent transfer streams
- Automatic Retry: Failed transfers automatically retry with only remaining files
- Progress Monitoring: Real-time progress bars, transfer speeds, and ETA
- Incremental Transfers: Skips files that already exist with matching sizes on the remote
- Adaptive Buffering: Configurable buffer sizes with mbuffer for optimal performance
- Comprehensive Logging: Detailed transfer logs with timestamps for analysis
pip install git+https://github.com/milmillin/copyem.gitgit clone https://github.com/milmillin/copyem.git
cd copyem
uv sync- Python 3.9+
- SSH access to remote host
tar,mbuffer, and standard Unix utilities on both local and remote systems
Transfer a directory to a remote host:
copyem /path/to/source user@remote.host /path/to/destinationcopyem /path/to/source user@remote.host /path/to/destination \
--parallel 4 \ # Use 4 parallel transfers
--buffer-size 2G \ # 2GB buffer per transfer
--speed 100M \ # Assume 100MB/s network speed
--include "*.txt" \ # Only transfer .txt files
--max-retries 5 \ # Retry failed transfers up to 5 times
--retry-delay 5.0 # Wait 5 seconds between retriessrc_dir: Source directory to copyremote: SSH remote (e.g., username@hostname.com)dst_dir: Target directory on remote
--include PATTERN: Include files matching this pattern (e.g., '.txt', '**/.py')-s, --speed SIZE: Assumed network speed for scheduling (default: 20M)-l, --latency SECONDS: File loading latency assumption (default: 0.05)-b, --buffer-size SIZE: Buffer size for transfers (default: 1G)-p, --parallel N: Number of parallel transfers (default: 1)-r, --max-retries N: Maximum retry attempts for failed transfers (default: 3)--retry-delay SECONDS: Delay between retry attempts (default: 2.0)--poll-interval SECONDS: Polling interval for monitoring (default: 0.5)
Size arguments support the following units:
B: BytesK/KB: Kilobytes (1024 bytes)M/MB: Megabytes (1024^2 bytes)G/GB: Gigabytes (1024^3 bytes)T/TB: Terabytes (1024^4 bytes)
Examples: 100M, 1.5G, 512K
Copyem first scans the source directory to identify all files that need to be transferred. It then queries the remote system to check which files already exist and have matching sizes, avoiding unnecessary transfers.
Files are distributed across parallel transfer streams using an optimized scheduling algorithm that:
- Minimizes overall transfer time
- Accounts for file size and network latency
- Balances load across parallel streams
Each transfer uses a pipeline architecture:
tar (create archive) | mbuffer (buffering) | ssh (network) | tar (extract)
This approach:
- Reduces per-file overhead
- Enables efficient buffering
- Maintains file permissions and metadata
The terminal UI provides real-time feedback:
- Individual transfer speeds and buffer status
- Overall progress bar with percentage complete
- Current and average transfer speeds
- Estimated time remaining
If a transfer fails:
- SSH output is analyzed to identify successfully transferred files
- Only remaining files are retried
- Progress is preserved across retries
- Failed transfers are reported in the final summary
During transfer, copyem displays:
[Starting transfer messages and file discovery...]
[mbuffer-1] in @ 38.0 MiB/s, out @ 38.0 MiB/s, 980 MiB total, buffer 82% full
[mbuffer-2] in @ 38.0 MiB/s, out @ 38.0 MiB/s, 980 MiB total, buffer 82% full
Curr: 83.0 MB/s | Avg: 78.5 MB/s | Time: 00m45s | ETA: 01m23s
[######------------] 45.2% (2.3GB/5.1GB)
After completion, a detailed summary is displayed:
============================================================
TRANSFER SUMMARY
============================================================
Transfer Statistics:
Total time: 02m15s
Files transferred: 1523/1523
Data transferred: 5.12 GB (5,497,558,528 bytes)
Effective speed: 38.52 MB/s
Transfer Status:
Successful transfers: 4/4
Failed transfers: 0/4
============================================================
ALL TRANSFERS COMPLETED SUCCESSFULLY
============================================================
Copyem creates detailed log files (copyem_YYYYMMDD_HHMMSS.log) containing:
- Timestamps for each file transfer
- Transfer stream identifiers
- File paths
These logs can be analyzed using the included visualization script in metrics/visualize_latency.py.
- Buffer Size: Larger buffers (1-4GB) generally improve performance for fast networks
- Parallel Transfers: Use 2-8 parallel transfers for optimal throughput
- Network Speed: Set
--speedto slightly below your actual network capacity - File Patterns: Use
--includeto filter files and reduce scanning time
__init__.py: Main entry point, command-line interface, and transfer orchestrationcore.py: File discovery, scheduling algorithm, and transfer pipeline setuplogger.py: Terminal UI management, progress tracking, and loggingutils.py: Utility functions for size parsing and formatting
- Retry Logic: Intelligent retry mechanism that tracks completed files via SSH output
- Progress Persistence: Transfer progress is maintained even when retrying failed transfers
- Resource Management: Proper cleanup of processes, file handles, and temporary files
- Thread Safety: Concurrent operations are properly synchronized
# Clone the repository
git clone https://github.com/milmillin/copyem.git
cd copyem
# Install with development dependencies
uv pip install -e ".[dev]"# You can run the transfer if you have remote hosts and test files
copyem /test/source user@host /test/dest --parallel 2The metrics/visualize_latency.py script can analyze log files to visualize:
- Inter-arrival times between files
- Transfer latencies between parallel streams
- Performance bottlenecks
Contributions are welcome! Please feel free to submit a Pull Request.
Milin Kodnongbua mil.millin@hotmail.com