Prepending data

[sleighsoft]

Is it possible or can you guide me on how I could prepend data to a trace instead of appending it?

Any help is greatly appreciated :)

[huww98]

You cannot prepend data currently, and it would be non-trivial to implement that.

Why do you want to prepend data, since the X-axis always represents time in this library? Maybe you want a right-to-left layout?

[sleighsoft]

I would like to load a particular slice of timing data and when panning left and right load and add additional timing data from either before or after. Ideally I would also be able to unload part of the timing data if whatever is currently loaded exceeds a memory limit.

I am thinking about using this library to render seismic data. Though 1 hour of data can easily consist of 700k or more points.
I therefore thought it may be better to incrementally load and unload data (see #6) while keeping the viewport fixed. Any better ideas are very much welcome :)

[Venryx]

I have the same request and a similar use-case: My program lets the user review eeg-data from a whole night of sleeping, from a device that records 512 values per second. For 8 hours of sleep, this means: 512*60*60*8 = 14,745,600 values

Even if TimeChart can handle loading that many values all at once, the problem is that I also need to perform processing on those values before displaying them (normalization, smoothing, and adding of two additional lines for derivative values), and this processing takes a significant amount of time.

I therefore want to apply the processing only for slices of the data that the user is currently viewing. Since the user may start viewing a section of data in the middle (and then pan either left or right), this means the charting library would need to either support prepending data, or editing entry values.

If prepending data is too difficult, would adding support for data-point editing be easier?

That functionality would open up two solutions:

1. Start by sending it an array for the whole night, but filled with simple 0s -- and as the user pans to different areas, I replace those 0s with the actual raw and processed values. This avoids calling the heavy processing code for sections of the night data not yet panned to, and should be sufficient so long as TimeChart can handle being sent an initial array of ~10 million zeroes.
2. Send TimeChart a small array, with, say, one data-point per pixel of the chart width. Then, as the user pans (I guess this would be handled by custom code rather than TimeChart), we dynamically update the ~1000 entries' values to reflect the slice of data currently panned to. Here is a demo of this in webgl-plot. (use the "Reverse" checkbox to toggle between "panning" left and right)

I would consider just using webgl-plot, since it has the entry-editing functionality I need; however, the major drawback is that webgl-plot appears to have no grid/axes rendering, which is an important aspect for my use-case.

[huww98]

I'm considering using Proxy to capture the changes to the data array so that every change you do to the data array will be reflected on the chart. Vue3 uses the same technique to detect changes in data and sync them to view.

But now I'm busy with my school works. I only have time to implement this after 1 or 2 weeks.

[Venryx]

Great to hear!

In the meantime, I did find a library that is able to handle 100k points, while maintaining good performance (and being able to add/edit data after initial render): Plotly.js

At first I thought the performance would not be good enough, because the standard 2d charts use SVG elements rather than WebGL, but I was shocked to find that it worked very smoothly, even when increasing the sample count to 100k, and panning/zooming in realtime: https://codepen.io/Venryx/pen/eYzpEzN

This is apparently because of the automatic entry "decimation" feature, whereby it simplifies the data to the current view (ie. if you're zoomed far out, you don't need to render a line between every point, but only every X points): https://community.plotly.com/t/polyline-simplification/1600

Because I've found a suitable library for my needs (zooming/panning + modify data anytime + good performance), I'm not in urgent need of the changes suggested above. (and @sleighsoft you should check it out too, starting with the demo I linked)

Entry editing still seems like a nice feature to add for this library at some point, but we have an alternative now, so no rush. :)

[leeoniya]

plotly is not so fast ;)

https://github.com/leeoniya/uPlot#performance

[Venryx]

Well, the benchmark there is using the WebGL renderer -- which is faster per line rendered (than the SVG-mode my demo uses Plotly in), but apparently isn't compatible with freeform array modifications. (when I switch to "scattergl" in my demo, the chart goes blank after pressing the "Extend data" button)

If the benchmark was using Plotly in SVG-renderer mode, it would be even slower -- in terms of render-time per line. However, because svg-mode Plotly comes with that data decimator/simplifier system (based on current pan/zoom), the demo I linked shows how it manages to still give good performance in the end, but without the restrictions that the webgl libraries currently have. (eg. it lets you add and edit entries at will)

Anyway, the best scenario would be a library that combines webgl-rendering with a solid data decimation/simplifier system; but in the meantime, Plotly (in SVG mode, with that option enabled) gets the job done sufficiently.

[huww98]

I personally don't like a universal data simplifier system. If you have more data points than your monitor resolution, you can set the line width to less than 1 pixels, and get details from rendered pixel opacity.

In the meantime, a domain-specific simplifier can be useful. For example, show the average value in each hour, and inform the end-user how these values are calculated.

[leeoniya]

@Venryx i would be interested to see how your dataset & use-case performs in uPlot. avoiding svg is a pretty straightforward win, but using webgl instead of plain canvas2d does not have the same clear benefit. for non-streaming, data exploration type cases, canvas2d is often the better choice.

[Venryx]

@leeoniya Yes, I've been considering uPlot. The main reason I'm going with Plotly for now is that the smoothness of the panning/scrolling impressed me. uPlot isn't bad either, but it's not quite as smooth as Plotly (with its data-decimation option on).

Compare:

1. uPlot (animation presumably equivalent to panning): https://leeoniya.github.io/uPlot/demos/stream-data.html
2. Plotly, with data-decimation (use pan tool): https://codepen.io/Venryx/pen/eYzpEzN

(Granted, the uPlot demo is updating three charts on the same page; it would be nice if there was a standalone page or panel for those demo charts, to confirm that the performance difference isn't just from that.)

Anyway, uPlot isn't bad, and would probably be workable in my app -- but the extra smoothness of Plotly is noticeable.

Of course, since it's accomplishing that smoothness merely through data-decimation, it should be transferable to the other charting libraries. However, from what I understand, few of them have such an option integrated. (or if they do, they are half-baked, such as the one for Chart.js, which I never got to work right)

[leeoniya]

the jumpiness is due to the data being updated & re-redered at 100ms intervals, not a perf limitation:

https://github.com/leeoniya/uPlot/blob/6ac8f395119ad5ee2eaf31932ab30027bb7025a4/demos/stream-data.html#L64

there's also no animation/transition support in uPlot, which i guess doesn't help the situation, but it's not a perf issue.

the plotly demo uses a hard-coded [-1000, 1000] y range, while uPlot has to re-find it for each series on each data update. you can config uPlot to avoid this overhead as well, though it's not enormous. anyways, the two examples are pretty apple-to-oranges, while the bench i linked is more apples-to-apples.

[Venryx]

Okay, that's good to hear; I'll likely give uPlot a try, if I hit significant annoyances or perf-issues while trialing Plotly.

By the way, it would be nice if the uPlot demos had some way for users to modify/sandbox them (a simple link to a codesandbox equivalent, for example).

That's something I've grown accustomed to when trying out libraries (especially for things like charting libraries, where there are literally hundreds of options), so when a site has demos that I can't edit, it makes me hesitant to evaluate them as closely. (for example, I might have noticed the 100ms update interval that way)

[Venryx]

By the way, I just took the time to figure out how to use "Local Overrides" in the Chrome dev-tools: https://www.afasterweb.com/2018/04/19/using-local-overrides-in-devtools

With it, I am now able to modify random online web demos, persistently, even if the demo doesn't have a jsfiddle/codesandbox/codepen page linked. :)

Anyway, I was thus able to try changing the interval to 10ms, and it is indeed a lot smoother now -- making it the next library I'd try.

[leeoniya]

a simple link to a codesandbox equivalent, for example).

i prefer to keep all the demo code locally and not rely on third party services + keeping external demos updated. i might do something like what i did with domvm in the future, though: https://domvm.github.io/domvm/demos/. i've been waiting on CodeMirror 6 to become stable first, and i'm currently blocked on codemirror/dev#284.

By the way, I just took the time to figure out how to use "Local Overrides" in the Chrome dev-tools: https://www.afasterweb.com/2018/04/19/using-local-overrides-in-devtools

that's pretty neat.

Anyway, I was thus able to try changing the interval to 10ms

anything less than 16.6666ms is pure waste, since that's a 60fps frame :) you're better off scheduling data updates via requestAnimationFrame()

[Venryx]

By the way, I put together a sandbox demo based on the uPlot mouse-wheel demo, and it works well even with 100k points: https://codepen.io/Venryx/pen/vYKNdZw

Make sure to use the middle mouse button to zoom and scroll, not the left-mouse-button zoom-selection. (as that is misconfigured to change the y-scaling, for some reason)

Anyway, I think your readme page comment on panning is too modest:

No built-in drag scrolling/panning. Maintaining good perf with huge datasets would require a lot of extra code & multiple elements to avoid continuous redraw and rescaling on each dragged pixel. If you have fewer than tens of thousands of datapoints, you can use uPlot's API to implement smooth zooming or panning. e.g. zoom-wheel, zoom-touch. Pagination of data also works well.

The performance seems good to me, in that demo with 100,000 points -- so I don't know why you set the threshold so low at "fewer than tens of thousands of datapoints".

[leeoniya]

Anyway, I think your readme page comment on panning is too modest:

under-promise, over-deliver :D

[Venryx]

under-promise, over-deliver :D

Okay, but you should link to a demo page showing off your under-promise, over-delivering then. :)

For example, uPlot is just as responsive as the Plotly demo now:
uPlot, 100k points: https://codepen.io/Venryx/pen/qBNOoYO
Plotly, 100k points: https://codepen.io/Venryx/pen/eYzpEzN

I wouldn't have expected that based on the readme contents...

[Venryx]

Also, I should probably stop spamming this TimeChart issue with uPlot-related information...

I made a new thread for it here: leeoniya/uPlot#329

I'll add further uPlot-related comments there from now on. (sorry, @huww98!)

[huww98]

That's OK.

I have a look at Plotly. it uses a clever strategy to handle user interaction. It just transforms the whole plot in response to the wheel event. and rerender the plot only once after the user interaction finished. By contrast, TimeChart rerenders every data point at every frame. I have to admit that although the strategy used by Plotly is not perfect, it works well enough.

[sleighsoft]

@Venryx I know Plotly, I added the prependData feature to its python backend a couple of weeks ago ;)

[huww98]

Hi all, finally I have implemented this. From v1.0.0-beta6, just use data.unshift({x, y}) to prepend data. Refer to the doc for more.