Skip to content

v0.1.33 - Auto-Reload Optimization with mtime Change Detection

Choose a tag to compare

View all tags
@quanhua92 quanhua92 released this 20 Nov 13:37
· 739 commits to main since this release
v0.1.33
3fbc2bf

🚀 Performance Optimization

This release introduces mtime-based change detection for auto-reload workers, dramatically reducing disk I/O while maintaining cache freshness.

✨ New Features

  • Smart file change detection: Auto-reload workers now track file modification times (mtime) and skip reading unchanged CSV files
  • 99.998% I/O reduction: From ~1.6 billion records/day to ~36K records/day
  • Comprehensive documentation: New docs/SYNC_LOAD.md explaining the sync and auto-reload architecture

📊 Performance Improvements

Before (v0.1.32):

  • Auto-reload: Read ALL files every 30 seconds
  • Daily I/O: ~555K records × 2,880 checks = 1.6 billion records
  • No distinction between active trading and idle periods

After (v0.1.33):

  • Auto-reload: Only read files whose mtime has changed
  • Trading hours: ~36K records (only recently synced files)
  • Off-hours/holidays: 0 records (no sync activity = no changes)
  • Reload time: 422ms → 5ms when files unchanged (98.8% faster)

🔧 Technical Details

How It Works:

  1. Track each CSV file's modification time after reading
  2. Before next reload: Check if current_mtime != stored_mtime
  3. If unchanged: Skip reading (file hasn't been modified by sync workers)
  4. If changed: Read file and update stored mtime

Key Insight:

  • NOT a time threshold check ("is file recent?")
  • IS a change detection check ("has mtime changed since last reload?")
  • Sync workers always update mtime when writing → reliable change detection

📝 Changes

Modified:

  • src/services/data_store.rs: Add mtime tracking and comparison logic
    • New field: file_mtimes: RwLock<HashMap<(String, Interval), SystemTime>>
    • Modified: load_interval() to check mtime before reading
    • Added: Detailed logging for files read vs skipped

Added:

  • docs/SYNC_LOAD.md: Comprehensive architecture documentation
    • Two-tier system: Sync workers (write) + Auto-reload workers (read)
    • Why sync always rewrites files (and why that's beneficial)
    • Timeline examples for trading hours and off-hours
    • FAQ and troubleshooting guide

🎯 Use Cases

This optimization is especially beneficial for:

  • ✅ 24/7 deployments: Zero I/O during market holidays/weekends
  • ✅ Resource-constrained VPS: Reduced CPU and disk usage
  • ✅ Long-running servers: Consistent low overhead regardless of market activity
  • ✅ Multiple data modes: Benefits both VN stocks and crypto data

📚 Documentation

See docs/SYNC_LOAD.md for detailed technical documentation including:

  • Architecture overview
  • Performance characteristics
  • Code references with line numbers
  • FAQ addressing common questions

🔗 Commits

  • b7e7a4f: Optimize auto-reload workers with file modification time checking
  • a56eaf5: Add comprehensive sync and auto-reload architecture documentation
  • 3fbc2bf: Bump version to 0.1.33
Assets 2
Loading