Repository navigation
What's Changed
Performance Improvements 🚀
- 8x faster snapshot operations through Cython-optimized Levenshtein distance calculations
- Snapshot time reduced from 20s to 1.6s for 10,000 unique values
- Check operations improved from 3.7s to 1.2s
Code Simplification 🧹
- Replaced complex DAFSA implementation with simple Python set
- Removed 60 lines of code while maintaining nearly identical performance
- More predictable memory usage across different data patterns
Technical Details
- Added Cython extension for Levenshtein distance computation
- Graceful fallback to pure Python if Cython unavailable
- Pre-compiled C files included for easier installation
- Compiler optimizations enabled (-O3, -march=native)
Compatibility
- Maintains zero-false-positive guarantee
- Backward compatible API
- Works with both Parquet and CSV files
Performance Benchmarks
| Operation | v0.1.0 | v0.1.1 | Speedup |
|---|---|---|---|
| Snapshot | 20s | 1.6s | 12.5x |
| Check | 3.7s | 1.2s | 3.1x |
Tested with 10,000 unique categorical values
Installation
pip install --upgrade driftfxNote: For best performance, ensure you have a C compiler available during installation to build the Cython extensions.
Contributors
- @chrisjwoodx
Full Changelog: v0.1.0...v0.1.1