When submission fails all attempts, Tick prepends the batch back only while there is room:
overCapacity := s.cfg.Buffer.Len(partitionKey)+len(tmp) >= s.cfg.Buffer.Capacity(partitionKey)
if !success && !overCapacity {
s.cfg.Buffer.PriorityPrepend(partitionKey, tmp)
}
(controlplane/telemetry/internal/telemetry/submitter.go:246-252)
The first time that check trips, the entire accumulated backlog goes rather than the oldest slice, and accumulation restarts from zero. Two problems follow:
- The samples that survive are worse than the ones that were lost. The account stores one
start_timestamp_microseconds and one sampling_interval_microseconds with a flat samples array (smartcontract/programs/doublezero-telemetry/src/state/device_latency_samples.rs:117-123) and the reader synthesizes each timestamp as start + i*interval (controlplane/telemetry/internal/data/device/latencies.go:205). A dropped run leaves no marker, so every sample appended afterwards is backdated by the full length of the gap.
- Dropping everything loses far more than it has to. Keeping the newest
Capacity - Len samples would bound the loss to what genuinely does not fit.
#4143 bounds -max-epoch-staleness to partitionBufferCapacity × -probe-interval with headroom, so the probe loop stops adding before this path is reached. That closes the route in but leaves the path itself sharp for any other caller or any future capacity change.
Fix: keep the newest Capacity - Len samples instead of discarding all of tmp, and log what was dropped. Note the slice handed to Recycle must stay the original one — reslicing from the middle would return a shifted array to the pool.
Reported by @ben-dz in review of #4143.
When submission fails all attempts,
Tickprepends the batch back only while there is room:(
controlplane/telemetry/internal/telemetry/submitter.go:246-252)The first time that check trips, the entire accumulated backlog goes rather than the oldest slice, and accumulation restarts from zero. Two problems follow:
start_timestamp_microsecondsand onesampling_interval_microsecondswith a flatsamplesarray (smartcontract/programs/doublezero-telemetry/src/state/device_latency_samples.rs:117-123) and the reader synthesizes each timestamp asstart + i*interval(controlplane/telemetry/internal/data/device/latencies.go:205). A dropped run leaves no marker, so every sample appended afterwards is backdated by the full length of the gap.Capacity - Lensamples would bound the loss to what genuinely does not fit.#4143 bounds
-max-epoch-stalenesstopartitionBufferCapacity × -probe-intervalwith headroom, so the probe loop stops adding before this path is reached. That closes the route in but leaves the path itself sharp for any other caller or any future capacity change.Fix: keep the newest
Capacity - Lensamples instead of discarding all oftmp, and log what was dropped. Note the slice handed toRecyclemust stay the original one — reslicing from the middle would return a shifted array to the pool.Reported by @ben-dz in review of #4143.