Skip to content

v0.7.0 — Frame-of-Reference + Bit Packing

Choose a tag to compare

@hussain-alsaibai hussain-alsaibai released this 28 Jun 09:16
· 14 commits to main since this release

What's New

Frame-of-Reference (FOR) Encoding

  • 4–8× memory reduction for bounded numeric columns (ages, scores, ratings)
  • Stores min once, bit-packs deltas into Python int bitmask
  • Auto-detects after sampling threshold (default 50 rows)
  • Auto-fallback when range exceeds 16 bits
  • Per-column via for_columns=[] on ColumnarTable
  • Transparent API — reads return full values, no changes needed
  • Update fallback to raw (same pattern as delta encoding)

Impact

Scenario Raw FOR Reduction
Ages 18-65 (100K rows) 400 KB ~88 KB 4.5×
Scores 0-100 (100K rows) 400 KB ~88 KB 4.5×

Tests

  • 6 new tests added, all passing
  • Zero regressions: 47 existing + 20 dict + 79 delta = 146 tests all pass

Closes