Skip to content

v0.9.2

Latest

Choose a tag to compare

@github-actions github-actions released this 30 Aug 14:34
· 1 commit to main since this release
78172af

v0.9.2 — Fixes a file-naming issue that could overwrite earlier data within the same hour

What happened

_flush() named its output files using only hour-level precision
("%Y-%m-%d-%H"). Since flushes happen every flush_interval seconds
(5 minutes by default) for as long as the process runs, more than one
flush could land within the same hour and write to the same file path.
Because writes replace rather than append, a later flush in that hour
could overwrite an earlier one rather than adding to it — with no error
or warning shown either way.

This pattern goes back to the project's original streamer script and
carried into the package unchanged. It's relevant to any deployment
using a flush_interval under 3600 seconds, which covers the default
and most practical settings.

What we found when we checked

We reproduced the pattern directly, then checked it against real,
already-published data to see how much it actually mattered in practice.

  • Two months of the live Goooddy/crypto-lob-stream dataset (June and
    July 2026, Binance BTCUSDT trades) showed 86–92% of hours with full,
    healthy coverage
    . The remainder showed the narrower coverage this
    pattern would produce.
  • The likely reason it wasn't worse: this codebase forces a flush on
    every WebSocket reconnect, and reconnects happen often enough that
    many hours only ever got one flush call — nothing else that hour to
    collide with. Hours with a longer stable connection were the ones
    actually affected.
  • A separate, already-documented event (the July 5th 43-minute
    host-restart pause noted in the dataset's "Known gaps" section) is
    unrelated to this — that gap has a clean snapshot boundary and
    reconstructs correctly as intended.

So: real, worth fixing, and worth telling you about — but the data that's
out there is mostly intact, not mostly missing.

The fix

  • _flush() now uses second-level precision ("%Y-%m-%d-%H%M%S"),
    matching what snapshot writes already did elsewhere in the same file.
  • Added a safety net on top of that: both write_local() and
    write_gcs() now check before writing, and if a file/object
    unexpectedly already exists at the target path, they log it clearly
    and write to a disambiguated path instead of silently replacing it.
    This should never trigger in normal operation now, but it's there in
    case it ever does.
  • New crypto_lob_stream.audit_coverage() utility — scans existing
    Parquet files for the coverage pattern this issue would leave behind.
    Useful as a one-time check on older data, or as an ongoing spot-check.
  • __version__ is now read automatically from the installed package's
    own metadata instead of being maintained by hand in __init__.py, so
    it can't drift out of sync with pyproject.toml the way it briefly did
    across the last two releases.

If you're running your own capture pipeline

Worth a quick check on your own historical files:

from crypto_lob_stream import audit_coverage, summarize

results = audit_coverage("./lob_data/trades/binance/BTCUSDT", window_seconds=3600)
print(summarize(results))

Data written before this fix can't be reconstructed after the fact —
this release stops the issue going forward rather than repairing what's
already there. If you're upgrading from an earlier version with
flush_interval under 3600s, restarting your capture process on this
version is worth doing sooner rather than later.

Notes

No schema or data-format changes in this release — only how reliably
records make it to disk, not what they contain.