Skip to content

v0.1.28 - CSV Performance Optimization

Choose a tag to compare

@quanhua92 quanhua92 released this 18 Nov 05:00
· 757 commits to main since this release

πŸš€ Performance Improvements

CSV Export Optimization (83-87% faster)

  • Pre-allocated String buffers - Eliminates reallocations during CSV generation
  • Replaced format!() with write!() - Reduces ~1.5M temporary allocations for large datasets
  • Added performance logging - Tracks CSV generation time for monitoring

Performance Results:

  • Daily CSV (71K records): 400-500ms β†’ 60ms (83-87% faster) βœ…
  • Single ticker minute: 0.20ms βœ…
  • 10 tickers minute: 2.07ms βœ…
  • Compression support: 2.4x size reduction (10MB β†’ 4.1MB with gzip)

Auto-Reload In-Memory Cache

  • Background workers now auto-reload expired cache entries
  • Prevents stale data from being served
  • Ensures cache stays fresh even when workers run in background

πŸ§ͺ Testing Improvements

  • Added compression testing with --compressed flag
  • Show both compressed and uncompressed metrics
  • Updated performance thresholds to realistic values (600ms/200ms/1200ms)
  • Fixed CSV line count test to trim whitespace from wc -l output

πŸ“ Technical Details

  • Total request time breakdown: cache lookup (~450ms) + CSV generation (60ms) + network overhead
  • The CSV generation itself is now highly optimized
  • All 17 integration tests passing βœ…

πŸ“Š Full Changelog

  • Optimize CSV export performance: 83% faster generation (da0cb02)
  • Fix CSV line count test: trim whitespace from wc -l output (48ede09)
  • Add background auto-reload for in-memory cache (4387e64)