v0.1.28 - CSV Performance Optimization
π Performance Improvements
CSV Export Optimization (83-87% faster)
- Pre-allocated String buffers - Eliminates reallocations during CSV generation
- Replaced
format!()withwrite!()- Reduces ~1.5M temporary allocations for large datasets - Added performance logging - Tracks CSV generation time for monitoring
Performance Results:
- Daily CSV (71K records): 400-500ms β 60ms (83-87% faster) β
- Single ticker minute: 0.20ms β
- 10 tickers minute: 2.07ms β
- Compression support: 2.4x size reduction (10MB β 4.1MB with gzip)
Auto-Reload In-Memory Cache
- Background workers now auto-reload expired cache entries
- Prevents stale data from being served
- Ensures cache stays fresh even when workers run in background
π§ͺ Testing Improvements
- Added compression testing with
--compressedflag - Show both compressed and uncompressed metrics
- Updated performance thresholds to realistic values (600ms/200ms/1200ms)
- Fixed CSV line count test to trim whitespace from
wc -loutput
π Technical Details
- Total request time breakdown: cache lookup (~450ms) + CSV generation (60ms) + network overhead
- The CSV generation itself is now highly optimized
- All 17 integration tests passing β