Performance & Scalability
Dynamic Thread Scaling: Optimized from fixed 8 threads to adaptive CPU-based scaling with --threads option
Thread-Local S3 Clients: Each thread gets its own client instead of sharing one client across all threads
Parallel Version Creation: Creating versions now happens in parallel instead of sequentially
Optimized Connection Pool: Added proper boto3 connection pool configuration and timeout settings
Memory Efficiency
On-Demand Data Generation: Changed from pre-generating all objects at once to streaming generation
Simple Data Option: Added --simple_data flag to reuse a single data object for all uploads
Incremental Progress: Avoids memory issues for large inputs like 10 million objects
Features
Checksum Options: Added --checksum to select from md5, crc32, crc32c, sha1, sha256, or none
S3 Configuration: Added proper S3 headers for different checksum algorithms
Debugging Control: Added --debug flag to control file logging
Real-time Statistics: Added progress reporting with upload rates and percentages
Other Improvements
Log Buffering: Reduced I/O overhead by batching log writes
Better Error Handling: Improved exception handling and user feedback
Streamlined Code: Cleaner organization and more efficient implementation
Input Validation: Better validation and feedback for user inputs