CSSTUDIO-3420 Waterfall Plot widget optimization: Remove data points that are not plotted.

[abrahamwolk]

This pull request implements the removal of data-points that are not plotted in the Waterfall Plot widget.

The motivation for this change is that for large waveform PVs (on the order of 10000 indices), I get OutOfMemory-exceptions when plotting the waveforms for 30 minutes to 1 hour.

A drawback of removing data-points is that detail may be lost when resizing the plot, which may occur either because the widget is resized or because one of the axes of the waterfall plot changes size (e.g., if an axis changes from being [0.0, 1.0, ..., 9.0] to [0.0, 1.0, ..., 10.0], requiring one more digit to be displayed).

[georgweiss]

Resize of the widget or change of axis in runtime should be a corner case, right?

[abrahamwolk]

I would not call it a corner case: if the Z-axis (represented using colors) is set to automatically resize based on the data-points that are received, one can expect resizing to occur, especially when starting the widget or when large fluctuations occur in the data.

The waterfall plot computes the point in time that a particular pixel in the plot corresponds to, and it plots the most recent data point before or at that point in time. Of course, when the dimensions of the plot are changed, then so are the points in time that each pixel in the graph corresponds to, and the most recent data point for each pixel will generally have changed. With this pull request, the most recent data point that was plotted before the resizing will be plotted instead. If a plot is resized several times, it will generally result in even more detail being lost. This is clearly not optimal.

The motivation for introducing this optimization is that a waveform of doubles with 10 000 indices updating at 10 times per second generates approximately 2,7 GB of data per hour. This kind of memory consumption is unreasonable, I think, even more so when considering waterfall plots spanning longer periods of time.

[georgweiss]

With resizing I was referring to runtime changes in the X/Y plane, but I get your point. That said, expecting the waterfall plot to handle a 10.000 element waveform at 10Hz is a bit unrealistic.

[abrahamwolk]

expecting the waterfall plot to handle a 10.000 element waveform at 10Hz is a bit unrealistic.

I don't think that's necessarily the case. If the data production is limited in order to accomodate graphing functionality in Phoebus, then the control system is being designed around UI considerations, which I think it shouldn't be. Also, if a 10 000 element waveform at 1Hz is plotted over 10 hours, that would still consume 2.7 GB of memory for that graph, which I think is not reasonable for a graph.

I do think that the widget should be able to handle any reasonable input data, and I also think there probably isn't one best solution since there are probably many solutions with different tradeoffs.

[jacomago]

Have you thought about using the waveform conversion algorithm that Joao has used for compressing the waveform to a display important waveform?

[abrahamwolk]

Have you thought about using the waveform conversion algorithm that Joao has used for compressing the waveform to a display important waveform?

No, I have not. I am not aware of this algorithm, is it documented somewhere?

[jacomago]

LTTB - Largest Triangle Three Bucket

Joao gave me:

https://www.base.is/flot/
https://github.com/sveinn-steinarsson/flot-downsample
https://github.com/parkertomatoes/lttb-cpp

[abrahamwolk]

@jacomago: It looks very interesting! However, I think it is outside the scope of this pull request.