[PROPOSAL] Introduce native jitter metrics (computed in streaming) for Component and e2e stream stability analysis

[GGraziadei]

Currently, Apache Storm provides comprehensive metrics for throughput and average latency (execute-latency, process-latency). However, in high-precision real-time systems, averages often mask critical performance instabilities.

This proposal introduces a native Jitter Metric calculated at two levels:

• Component level (Step Jitter): Measures the variance in execution time within individual Bolts and Spouts.
• Topology level (Global Jitter): Measures the variance in e2e completion latency for fully acked tuples.

In deterministic real-time processing, the variance of the latency is as important as the latency itself (https://ieeexplore.ieee.org/abstract/document/10877871).

Why analysing jitter matters for real-time

In deterministic real-time processing, predictability of latency is as important as latency itself. This is a constraint to building a deterministic system.

• Mcro-burst detection: high jitter reveals short spikes that average latency smooths out.
• Compliance: modern SLAs rely on percentiles (e.g., P99). Jitter is a strong leading indicator of tail-latency degradation.
• Root Cause Analysis: high component jitter means GC pressure or resource contention; instead, high global jitter with stable components suggests network congestion or shuffle bottlenecks.
• Bottleneck identification: jitter enables precise identification of where bottlenecks occur in the topology and helps distinguish their underlying causes, making performance issues easier to diagnose and resolve.

Proposed model: Exponentially Weighted Moving Average (EWMA)

To ensure negligible performance impact, I propose to use an Exponentially Weighted Moving Average (EWMA), following RFC 1889 logic https://www.rfc-editor.org/rfc/rfc1889#appendix-A.8

Mathematical Model:
J_new = J_old + (|D_current - D_previous| - J_old) / 16

GIVEN a State {ewmaJitter, lastTransit}
PROCEDURE addValue(transitMs)
    IF transitMs < 0 THEN 
        EXIT PROCEDURE

    IF lastTransit IS NOT UNINITIALIZED THEN
        // Calculate the absolute difference between the current and previous transit time
        deviation = ABS(transitMs - lastTransit)
        
        // Update the Exponentially Weighted Moving Average using the RFC 1889 smoothing factor
        ewmaJitter = ewmaJitter + (deviation - ewmaJitter) * RFC1889_ALPHA
    END IF

    // Store current transit time for the next iteration
    lastTransit = transitMs
END PROCEDURE

Performance impact

• Minimal computational overhead: by utilizing an EWMA, we avoid the need for storing large datasets or sliding window buffers. The jitter is updated via a single linear equation, requiring only basic arithmetic.
• Memory efficiency: The EWMA algorithm is extremely memory-light, requiring only a single persistent variable (8 bytes) per executor to maintain the moving average state, plus a reference for the previous latency sample.
• System calls: To eliminate redundant overhead, the metric hooks into existing latency tracking logic. This point requires additional brainstorming to evaluate already sampled metrics.

Limitations and constraints

• Clock skew: Global jitter may be affected in the case of unsynchronised nodes. However, since jitter measures variance between consecutive samples, constant skew cancels out mathematically.
• Sampling bias: Low sampling rates may miss high-frequency jitter spikes.
• Warm-up: as an EWMA-based metric, values may fluctuate initially before stabilizing.

[rzo1]

Thanks for the detailed write-up - the motivation is solid and the RFC 1889 EWMA is genuinely cheap, so the cost
side of the proposal is hard to argue with. A few things I'd like to see clarified before this moves to a PR:

1. Relationship to existing Dropwizard metrics. Storm's metrics v2 is built on Dropwizard, and the existing
   execute-latency / process-latency / complete-latency timers are already Histogram/Timer instances with reservoir
   sampling. That gives us P99/P999, stddev, and mean for free, and they flow through the existing reporters
   (Prometheus, Graphite, etc.). Could you spell out what EWMA-of-abs-diff gives operators that the histogram
   percentiles don't? The RFC 1889 form is a great fit for an RTP receiver computing inter-arrival jitter on the fly,
   but operators consuming Storm metrics typically want tail percentiles, and the two are not the same statistic. If
   the answer is "cheaper than a reservoir," it's worth quantifying - Dropwizard's exponentially decaying reservoir is
   also quite light.

2. Definition. "Jitter" gets read three different ways in practice: (a) RFC 1889 smoothed inter-arrival deviation,
   (b) latency standard deviation, (c) tail spread (P99 − P50). The proposal picks (a), which is fine, but the
   doc/metric name should be explicit so users don't assume (b) or (c), right?

3. Smoothing factor. α = 1/16 is tuned for RTP's packet cadence. Storm tuple rates span many orders of magnitude:
   at 1M tuples/sec α=1/16 averages over microseconds of wall clock, at 10/sec over seconds. This should be
   configurable per topology (or per component), not a constant.

4. Integration point. The line "hooks into existing latency tracking logic" is doing a lot of work. Could you sketch
   where exactly BoltExecutor / SpoutExecutor latency sampling paths, the stats BoltExecutorStats /
   SpoutExecutorStats, or a parallel listener? That's the part that determines whether this is a 50-line change or a
   much larger one, and it'd be good to align on it before any code lands.

5. Acked-only e2e jitter. Restricting global jitter to fully-acked tuples introduces a selection bias: failed and
   timed-out tuples are often exactly the ones with pathological latency. Worth either including them or documenting
   the bias clearly.

6. Clock skew. "Constant skew cancels mathematically" is true, but e2e measurements cross workers/nodes where the
   relevant problem is drift, not constant offset. NTP-synced clocks still drift on the order of ms, which can dominate
   the jitter signal at low latencies. Worth calling out as a real limitation rather than a dismissed one (at least in the documentation of such a feature)

Happy to help review a PR once the integration point and the EWMA-vs-histogram-percentile question are settled.

[GGraziadei]

Hello @rzo1 thank you for your comment.

1. My ultimate goal is to move towards a self-regulating system, one capable of detecting congestion in real-time and dynamically modulating throughput across different stages, increasing the determinism of the flow transmission. Any robust control system requires a feedback loop, and while this is a complex goal that will likely be achieved in multiple steps, the absolute first requirement is a reliable, high-frequency control signal. I fully recognise that similar metrics can be derived from existing histograms, which are excellent for post-hoc analysis and human-readable dashboards. However, what I am proposing is a metric calculated on-the-fly, directly within the stream of transit. There is a fundamental difference in how these signals behave: histograms with reservoirs are inherently batch-oriented or sampling-based; they require a window of data to be collected before a statistically significant percentile can be computed. This introduces a lag that is often too high for reactive control logic. RFC 1889 EWMA instead is calculated continuously (or with a very high sampling ratio). Every single tuple (or a high percentuage or tuples0 updates the state in constant time. This lack of sampling delay provides a reactive feedback signal that can detect micro-bursts or jitter the moment they occur. In short, while histograms tell us how the system performed, this EWMA signal tells us how the system is performing right now. This immediacy is what will eventually allow an executor to detect a bottleneck and propagate that signal upstream to stabilize ingestion and ensure a more deterministic flow.
2. That is a valid point. Since jitter is often overloaded, I will explicitly label this as RFC 1889 Inter-arrival Jitter (definition A) in the documentation. By naming the metric precisely (e.g., jitter_rfc1889_a), we can clearly distinguish this reactive control signal from the descriptive statistics (P99 or StdDev) already provided by Dropwizard.
3. I completely agree that the smoothing factor has to be a configurable parameter and not a constant.
4. According to point 1, the idea is to extend the BoltExecutor and SpoutExecutor to create two new classes (e.g. BoltDeterministicExecutor / SpoutDeterministicExecutor)
5. That is a great point regarding selection bias. Including failed tuples is necessary for a complete picture, but I want to avoid polluting the jitter signal with the pathological latencies typical of timeouts. I propose splitting this into two distinct metrics: one for successful tuples and one for failed/timed-out tuples. This provides full visibility without compromising the precision of the primary control signal.
6. I completely agree, it is required to add this limitation in the documentation.