Skip to content

v0.1.1

Choose a tag to compare

@nedpark nedpark released this 24 Oct 10:33
· 11 commits to main since this release
b7221ab

v0.1.1: High-Performance Columnar Engine (Apache Arrow/Polars model)

Introduce a typed columnar storage architecture inspired by Apache Arrow and Polars to eliminate cell-level DataValue boxing, reduce dictionary allocations, and dramatically accelerate core DataFrame operations.

Highlights:

  • Typed Columnar Storage: Added BitSet, AnyColumn protocol, and specialized columnar buffers (Float64Column, Int64Column, StringColumn, CategoricalColumn, BooleanColumn, DateColumn, AnyValueColumn) with zero-copy slicing.
  • 64-bit Word Bitmap (BitSet): Zero-overhead null tracking and fast boolean mask evaluation with hardware popcount.
  • High-Performance Joins: Replaced per-row dictionary construction with integer index gathering and slice(atOptional:) columnar slicing (~3x faster left joins: 19.59ms -> 6.95ms on 40,000 rows).
  • Vectorized GroupBy: Single-pass dense Int32 group ID assignment and scatter aggregation directly into contiguous accumulation buffers (43% faster: 12.01ms -> 6.79ms).
  • Direct Primitive Sorting: Fast paths for Double, categorical code ranks, timestamps, and strings directly sorting raw index buffers.
  • Streamlined CSV Parser: File size-based capacity pre-allocation, quote-free line splitting, UTF-8 BOM auto-detection, and zero-allocation Substring-based numeric parsing (130ms -> 84.39ms).
  • 100% Backward Compatibility: All 63 existing unit tests pass with zero compiler warnings."