ref(spans): Extract flush_segment pipeline helpers#116149
Conversation
d4c017a to
596579f
Compare
| max_flush_segments = options.get("spans.buffer.max-flush-segments") | ||
| max_segments_per_shard = math.ceil(max_flush_segments / shard_factor) | ||
|
|
||
| flush_candidates, load_ids_latency_ms = self._load_flush_candidates( |
There was a problem hiding this comment.
One tip to separate observability better:
create a FlushStats class with the following methods:
def add_load_ids_latency
def add_load_data_latency_ms
def add_decompress_latency_ms
def add_segments(segment_keys, segment_to_queue, payloads)
def flush()
Create the object at the beginning of the methods and pass it to the other funcitons.
Each of the functions contributes by adding observability data.
At the end of the method you flush it once.
Now the observability code is totally outside this method.
| flush_candidates: Sequence[FlushCandidate], | ||
| loaded_segment_data: LoadedSegmentData, |
There was a problem hiding this comment.
tip:
you are passing two sequences that are supposed to match 1:1. This generally introduces ways this method can break and additional conditions to verify, which increase complexity: what if the two lists do not match ?
This is one of the reasons dealing with the redis api is such a pain.
I'd recommend adjusting the data structures so you have to pass only one (maybe loadedSegmentData) and ensure it contains all the objects you need from flush_candidates.
Refs STREAM-1001
Refactors the
flush_segmentsto be an orchestration function with unit testable helpers:_load_flush_candidates: loading ready flush candidates from queue shards_acquire_locks_for_flush_candidates: acquiring per-segment flush locks_load_segment_data: loading payload keys and payload bytes_build_flushed_segments: building producer-readyFlushedSegmentobjects_record_segment_loss_metrics: recording segment loss / expiration metricsAlso adds/moves datamodels (
FlushCandidate,FlushedSegment,OutputSpan) intobuffer_types.pyalongside the existing span buffer value types.