Skip to content

v0.1.2

Latest

Choose a tag to compare

@vnvo vnvo released this 07 Feb 23:40

flowstats v0.1.2

Bug Fixes

ReservoirSampler: Fixed biased merge producing non-uniform samples

The merge() implementation treated each item in the other reservoir as a single observation, incrementing count by 1 per reservoir slot instead of accounting for the actual number of stream items each slot represents. This produced heavily biased samples when merging reservoirs that had processed different stream sizes.

The fix uses weighted random selection: each output slot is drawn from self with probability self.count / total_count and from other with probability other.count / total_count, correctly preserving uniform sampling over the combined stream. Empty and underfilled reservoirs are handled as special cases.

TDigest: Fixed quantile/rank interpolation producing inconsistent results

quantile_impl() and rank_impl() used different interpolation models — quantile used cumulative boundaries while rank used centroid midpoints — causing rank(quantile(q)) ≠ q roundtrip failures. Both methods now use a consistent piecewise linear model through midpoint anchors: (rank=0, value=min) → (rank=cumulative_i + weight_i/2, value=mean_i) → (rank=count, value=max).

TDigest: Fixed division-by-zero in interpolation with duplicate values

When consecutive centroids had equal means (common with duplicate-heavy data), denominators in interpolation fractions could be zero, producing NaN. All division points in quantile_impl() and rank_impl() are now guarded with if denom <= 0.0 { return safe_boundary; }.

TDigest: Fixed potential NaN from asin() input drift

The scale function computed asin(2*q - 1) without clamping. Floating-point arithmetic could push the argument slightly outside [-1, 1], producing NaN. The argument is now clamped: (2.0 * q - 1.0).clamp(-1.0, 1.0).

TDigest: Fixed merge rejecting effectively-equal compression parameters

Merge used f64::EPSILON (~2×10⁻¹⁶) to compare compression parameters, which rejected values that differed only due to serialization roundtrips. Now uses relative tolerance of 1×10⁻⁶.

Performance

TDigest: Eliminated clone-per-query overhead

quantile() and rank() previously cloned the entire TDigest and compressed the clone on every call when the buffer was non-empty. The internal structure now uses RefCell for lazy compression on &self query methods, while mutation methods (add, merge) use get_mut() for zero-overhead access.

Note: This makes TDigest Send but not Sync. For shared concurrent access, wrap in Arc<Mutex<_>> or Arc<RwLock<_>>.

Other Improvements

  • TDigest: Added compression parameter validation — TDigest::new() now panics on non-finite, zero, or negative compression values instead of silently producing broken digests.
  • TDigest: Made push() private — users should use add() from the QuantileSketch trait; the redundant push() method is no longer part of the public API.
  • TDigest: Improved num_centroids() documentation — clarifies that the count excludes values still in the uncompressed buffer.
  • README: Fixed incorrect default feature flags — now matches Cargo.toml (all algorithm families enabled by default).
  • README: Fixed CountMinSketch example — add() requires a count argument.
  • README: Added TDigest thread-safety note to the Parallel Processing section.