v0.9.2

v0.9.2 — Fixes a file-naming issue that could overwrite earlier data within the same hour

What happened

_flush() named its output files using only hour-level precision
("%Y-%m-%d-%H"). Since flushes happen every flush_interval seconds
(5 minutes by default) for as long as the process runs, more than one
flush could land within the same hour and write to the same file path.
Because writes replace rather than append, a later flush in that hour
could overwrite an earlier one rather than adding to it — with no error
or warning shown either way.

This pattern goes back to the project's original streamer script and
carried into the package unchanged. It's relevant to any deployment
using a flush_interval under 3600 seconds, which covers the default
and most practical settings.

What we found when we checked

We reproduced the pattern directly, then checked it against real,
already-published data to see how much it actually mattered in practice.

• Two months of the live Goooddy/crypto-lob-stream dataset (June and
  July 2026, Binance BTCUSDT trades) showed 86–92% of hours with full,
  healthy coverage. The remainder showed the narrower coverage this
  pattern would produce.
• The likely reason it wasn't worse: this codebase forces a flush on
  every WebSocket reconnect, and reconnects happen often enough that
  many hours only ever got one flush call — nothing else that hour to
  collide with. Hours with a longer stable connection were the ones
  actually affected.
• A separate, already-documented event (the July 5th 43-minute
  host-restart pause noted in the dataset's "Known gaps" section) is
  unrelated to this — that gap has a clean snapshot boundary and
  reconstructs correctly as intended.

So: real, worth fixing, and worth telling you about — but the data that's
out there is mostly intact, not mostly missing.

The fix

• _flush() now uses second-level precision ("%Y-%m-%d-%H%M%S"),
  matching what snapshot writes already did elsewhere in the same file.
• Added a safety net on top of that: both write_local() and
  write_gcs() now check before writing, and if a file/object
  unexpectedly already exists at the target path, they log it clearly
  and write to a disambiguated path instead of silently replacing it.
  This should never trigger in normal operation now, but it's there in
  case it ever does.
• New crypto_lob_stream.audit_coverage() utility — scans existing
  Parquet files for the coverage pattern this issue would leave behind.
  Useful as a one-time check on older data, or as an ongoing spot-check.
• __version__ is now read automatically from the installed package's
  own metadata instead of being maintained by hand in __init__.py, so
  it can't drift out of sync with pyproject.toml the way it briefly did
  across the last two releases.

If you're running your own capture pipeline

Worth a quick check on your own historical files:

from crypto_lob_stream import audit_coverage, summarize

results = audit_coverage("./lob_data/trades/binance/BTCUSDT", window_seconds=3600)
print(summarize(results))

Data written before this fix can't be reconstructed after the fact —
this release stops the issue going forward rather than repairing what's
already there. If you're upgrading from an earlier version with
flush_interval under 3600s, restarting your capture process on this
version is worth doing sooner rather than later.

Notes

No schema or data-format changes in this release — only how reliably
records make it to disk, not what they contain.

v0.9.1

Highlights

Patch release: fixes a crash in compact() introduced by 0.9.0's new exchange_ts field. Any bucket of files spanning the 0.9.0 upgrade — older files without exchange_ts alongside newer ones with it — would raise ArrowInvalid: Schema at index N was different when compacted, since compact() previously required byte-identical schemas across every input file. compact() now tolerates an added column across a bucket, filling it with null in the older rows rather than raising. A genuine type conflict on a column both schemas share still raises, as it should — this only relaxes the added-column case.

If you upgraded straight from a pre-0.9.0 version and haven't yet run compact()/compact_tree() over data spanning that boundary, this release is what makes that safe. No schema or behavior changes beyond compact() itself — 0.9.0's timestamp_ms/exchange_ts semantics are unchanged.

Full Changelog: v0.9.0...v0.9.1

v0.9.0

Highlights

Local receive time and exchange event time are now separate, honest fields. Previously, timestamp_ms was inconsistent — some record types on some exchanges carried the exchange's own timestamp, others carried local receive time, with no way to tell which from the data itself. From this release forward, timestamp_ms is always local receive time, and a new nullable exchange_ts field carries the exchange's own event time separately, where the exchange provides one — never silently backfilled with local time when it doesn't.

Beyond correctness, this opens a research angle that wasn't cleanly available before: timestamp_ms - exchange_ts gives an approximate per-exchange, per-message-type capture latency, directly from the data already being collected.

See the README's new "Timestamps" section for full details, and the migration note there if you're working with data captured before this release.

v0.8.0

🚀 v0.8.0 - LOB Reconstruction, File Compaction & Advanced Futures Data
This release transforms crypto-lob-stream from a raw data ingestion script into a complete data engineering toolkit for market microstructure research.

📈 Advanced Futures / Perpetuals Data
In addition to funding rates, the futures adapters (binance_futures, okx_swap, bybit_linear) now capture two critical new tables for liquidity stress research:

liquidations/: Captures forced position closures in real-time (the "fire").

open_interest/: Captures total outstanding leveraged exposure (the "fuel").

🛠️ New Tool: LOB Reconstruction & The "Ghost-Level" Fix
Reconstructing an order book from diffs without tracking depth limits leads to "ghost levels" (prices that fall out of scope but are never explicitly zeroed out by the exchange).

Added reconstruct(): A new built-in helper that loads offline Parquet snapshots/diffs, replays them, and actively prunes the book to the correct exchange-specific depth.

🗜️ New Tool: Parquet File Compaction
Continuous streaming creates thousands of tiny 5-minute Parquet files, which slows down querying (especially in cloud storage).

Added compact() and compact_tree(): Easily merge small leaf files into daily, weekly, monthly, or yearly chunks without losing data fidelity.

⚠️ Reminder on Output Structure (Since 0.7.0)
All Parquet tables now carry an explicit exchange column, and paths are nested as {prefix}/{exchange}/{asset}/.... If you have legacy data from 0.6.x, move it under an {exchange}/ subfolder manually to unify your data tree.

v0.7.0

⚠️ Breaking change

Output directory layout changed: every table now nests one level deeper under {exchange}/, and every Parquet table (trades, depth, snapshots) gained a new exchange column.

• Before: {output_dir}/trades/{asset}/...
• Now: {output_dir}/trades/{exchange}/{asset}/...

Existing 0.6.x data won't merge automatically with new data — move it under an {exchange}/ subfolder manually if you want one unified tree.

New features

Multi-exchange streaming — run multiple exchanges concurrently in one process/event loop:

LOBStreamer(exchanges=[
    {"exchange": "binance", "assets": ["BTCUSDT"]},
    {"exchange": "kraken", "assets": ["BTC/USD"]},
]).run()

or --exchanges "binance:BTCUSDT;kraken:BTC/USD" on the CLI.

Gap detection (detect_gaps=True, on by default) — flags sequence-id discontinuities for exchanges with real sequence numbers (binance, binance_futures, bybit, bybit_linear, okx, okx_swap), with automatic re-snapshot on a detected gap where a REST endpoint exists. New gaps/ table.

Checksum verification (verify_checksums=True, off by default) — for Kraken, maintains a live order-book mirror and verifies it against Kraken's CRC32 checksum. New checksums/ table.

Futures/perps support — 3 new exchange adapters:

• binance_futures — Binance USDⓢ-M perpetuals (depth, trades, funding rate)
• okx_swap — OKX USDT-margined perpetual swaps (depth, trades, funding rate)
• bybit_linear — Bybit USDT perpetuals (depth, trades, funding rate + mark price)

All three write to a new funding/ table.

Bug fixes (found via live testing, not review)

• Kraken checksum: local order-book mirror wasn't truncated to the subscribed depth, causing a 100% mismatch rate against live data.
• Binance Futures: the funding-rate stream was silently dropped — Binance now routes /market channels (like markPrice) separately from /public ones, and the unrouted connection only got the latter. Fixed via a second, /market-routed connection.
• Bybit (and bybit_linear): gap detection used the wrong counter for continuity (seq, a cross-venue sequence on a different numeric scale, instead of u, the real per-symbol update id), causing constant false-positive gaps. Predates this release — affected plain bybit too, just hadn't been exercised by gap detection until now.

Known limitations

• okx_swap funding records have mark_price=None (OKX's mark price needs a second, unverified channel — documented rather than guessed)
• Auto-resync-on-gap only works for REST-snapshot exchanges; OKX/Bybit-family exchanges log the gap but need a reconnect for a fresh snapshot
• Kraken Futures and Coinbase's institutional perpetuals (Coinbase International Exchange) aren't covered — both are architecturally separate platforms from the spot APIs already integrated, not a simple addition
• Dated/quarterly futures contracts aren't supported, only perpetuals.