Skip to content

SparkPL v2.0.0 - Pure Polars-Spark Conversion (Pandas-Free)

Choose a tag to compare

@Boadzie Boadzie released this 02 Jul 10:01
· 9 commits to main since this release

Major release removing pandas dependency and implementing direct Arrow conversion for optimal performance.
🚀 Breaking Changes

Removed all pandas dependencies - package now works without pandas installed
Updated API to use pure Polars ↔ Spark conversion methods

✨ New Features

Direct Arrow conversion using toArrow() and createDataFrame(arrow_table) (Spark 4.0+)
Native fallback methods for older Spark versions
Structured logging with loguru integration
Batch processing for large datasets
Enhanced error handling with custom exceptions

⚡ Performance Improvements

Faster conversions using native Arrow integration
Reduced memory footprint without pandas overhead
Optimized type mapping between Polars and Spark