v0.9.2 — Fixes a file-naming issue that could overwrite earlier data within the same hour
What happened
_flush() named its output files using only hour-level precision
("%Y-%m-%d-%H"). Since flushes happen every flush_interval seconds
(5 minutes by default) for as long as the process runs, more than one
flush could land within the same hour and write to the same file path.
Because writes replace rather than append, a later flush in that hour
could overwrite an earlier one rather than adding to it — with no error
or warning shown either way.
This pattern goes back to the project's original streamer script and
carried into the package unchanged. It's relevant to any deployment
using a flush_interval under 3600 seconds, which covers the default
and most practical settings.
What we found when we checked
We reproduced the pattern directly, then checked it against real,
already-published data to see how much it actually mattered in practice.
- Two months of the live
Goooddy/crypto-lob-streamdataset (June and
July 2026, Binance BTCUSDT trades) showed 86–92% of hours with full,
healthy coverage. The remainder showed the narrower coverage this
pattern would produce. - The likely reason it wasn't worse: this codebase forces a flush on
every WebSocket reconnect, and reconnects happen often enough that
many hours only ever got one flush call — nothing else that hour to
collide with. Hours with a longer stable connection were the ones
actually affected. - A separate, already-documented event (the July 5th 43-minute
host-restart pause noted in the dataset's "Known gaps" section) is
unrelated to this — that gap has a clean snapshot boundary and
reconstructs correctly as intended.
So: real, worth fixing, and worth telling you about — but the data that's
out there is mostly intact, not mostly missing.
The fix
_flush()now uses second-level precision ("%Y-%m-%d-%H%M%S"),
matching what snapshot writes already did elsewhere in the same file.- Added a safety net on top of that: both
write_local()and
write_gcs()now check before writing, and if a file/object
unexpectedly already exists at the target path, they log it clearly
and write to a disambiguated path instead of silently replacing it.
This should never trigger in normal operation now, but it's there in
case it ever does. - New
crypto_lob_stream.audit_coverage()utility — scans existing
Parquet files for the coverage pattern this issue would leave behind.
Useful as a one-time check on older data, or as an ongoing spot-check. __version__is now read automatically from the installed package's
own metadata instead of being maintained by hand in__init__.py, so
it can't drift out of sync withpyproject.tomlthe way it briefly did
across the last two releases.
If you're running your own capture pipeline
Worth a quick check on your own historical files:
from crypto_lob_stream import audit_coverage, summarize
results = audit_coverage("./lob_data/trades/binance/BTCUSDT", window_seconds=3600)
print(summarize(results))Data written before this fix can't be reconstructed after the fact —
this release stops the issue going forward rather than repairing what's
already there. If you're upgrading from an earlier version with
flush_interval under 3600s, restarting your capture process on this
version is worth doing sooner rather than later.
Notes
No schema or data-format changes in this release — only how reliably
records make it to disk, not what they contain.