Skip to content

v0.1.1: 8x Performance Boost with Cython

Latest

Choose a tag to compare

@christopherhwood christopherhwood released this 09 Jul 09:29
· 1 commit to main since this release

What's Changed

Performance Improvements 🚀

  • 8x faster snapshot operations through Cython-optimized Levenshtein distance calculations
  • Snapshot time reduced from 20s to 1.6s for 10,000 unique values
  • Check operations improved from 3.7s to 1.2s

Code Simplification 🧹

  • Replaced complex DAFSA implementation with simple Python set
  • Removed 60 lines of code while maintaining nearly identical performance
  • More predictable memory usage across different data patterns

Technical Details

  • Added Cython extension for Levenshtein distance computation
  • Graceful fallback to pure Python if Cython unavailable
  • Pre-compiled C files included for easier installation
  • Compiler optimizations enabled (-O3, -march=native)

Compatibility

  • Maintains zero-false-positive guarantee
  • Backward compatible API
  • Works with both Parquet and CSV files

Performance Benchmarks

Operation v0.1.0 v0.1.1 Speedup
Snapshot 20s 1.6s 12.5x
Check 3.7s 1.2s 3.1x

Tested with 10,000 unique categorical values

Installation

pip install --upgrade driftfx

Note: For best performance, ensure you have a C compiler available during installation to build the Cython extensions.

Contributors

  • @chrisjwoodx

Full Changelog: v0.1.0...v0.1.1