v0.8.0
🚀 v0.8.0 - LOB Reconstruction, File Compaction & Advanced Futures Data
This release transforms crypto-lob-stream from a raw data ingestion script into a complete data engineering toolkit for market microstructure research.
📈 Advanced Futures / Perpetuals Data
In addition to funding rates, the futures adapters (binance_futures, okx_swap, bybit_linear) now capture two critical new tables for liquidity stress research:
liquidations/: Captures forced position closures in real-time (the "fire").
open_interest/: Captures total outstanding leveraged exposure (the "fuel").
🛠️ New Tool: LOB Reconstruction & The "Ghost-Level" Fix
Reconstructing an order book from diffs without tracking depth limits leads to "ghost levels" (prices that fall out of scope but are never explicitly zeroed out by the exchange).
Added reconstruct(): A new built-in helper that loads offline Parquet snapshots/diffs, replays them, and actively prunes the book to the correct exchange-specific depth.
🗜️ New Tool: Parquet File Compaction
Continuous streaming creates thousands of tiny 5-minute Parquet files, which slows down querying (especially in cloud storage).
Added compact() and compact_tree(): Easily merge small leaf files into daily, weekly, monthly, or yearly chunks without losing data fidelity.
All Parquet tables now carry an explicit exchange column, and paths are nested as {prefix}/{exchange}/{asset}/.... If you have legacy data from 0.6.x, move it under an {exchange}/ subfolder manually to unify your data tree.