Repository navigation
SparkPL v2.0.0 - Pure Polars-Spark Conversion (Pandas-Free)
Major release removing pandas dependency and implementing direct Arrow conversion for optimal performance.
🚀 Breaking Changes
Removed all pandas dependencies - package now works without pandas installed
Updated API to use pure Polars ↔ Spark conversion methods
✨ New Features
Direct Arrow conversion using toArrow() and createDataFrame(arrow_table) (Spark 4.0+)
Native fallback methods for older Spark versions
Structured logging with loguru integration
Batch processing for large datasets
Enhanced error handling with custom exceptions
⚡ Performance Improvements
Faster conversions using native Arrow integration
Reduced memory footprint without pandas overhead
Optimized type mapping between Polars and Spark