-
Notifications
You must be signed in to change notification settings - Fork 91
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix ArcticDB reading streaming data #1647
Conversation
8204c01
to
14bae62
Compare
return decode_timeseries_descriptor(hdr, data, begin, end, segment.descriptor()); | ||
} | ||
|
||
std::optional<TimeseriesDescriptor> decode_timeseries_descriptor_for_incompletes( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We could use more abstract naming here as it's a bit weird to refer to incompletes. Like decode_timeseries_descriptor_using_header_stream_descriptor
. I settled on the current naming as it makes the purpose of the weird special case clearer for people changing the codec
layer (it's a good hint about what they might break).
Please see the description on the `master` version of this change, #1647 The implementation is significantly different to the master version, because master is written on top of the new binary encoding work. --------- Co-authored-by: Alex Seaton <alex.seaton@man.com>
Reference Issues/PRs
Fixes https://github.com/man-group/arcticdb-man/issues/80
What does this implement or fix?
Allows ArcticDB to read incomplete segments written by arcticc tick collectors.
Any other comments?
arcticc has a different append incompletes logic to ArcticDB. It writes a dummy
StreamDescriptor
on theTimeseriesDescriptor
, and relies entirely on theStreamDescriptor
in the segment header. That approach is better than the existing ArcticDB approach, which writes the sameStreamDescriptor
twice (directly in the header, and in theTimeseriesDescriptor
).ArcticDB was reading this dummy stream descriptor and crashing.
Firstly, this PR changes readers to first check the StreamDescriptor on the header of incompletes, rather than using the one stamped on the
TimeseriesDescriptor
. It also adds a backwards compat test for the format.Secondly, and this is not required for the PR to be logically correct, we change writers so that, like arcticc, they do not duplicate the
StreamDescriptor
. This is done in 99d0870.Checklist
Checklist for code changes...