Replies: 7 comments 4 replies
-
|
Does the xArrayLinSorted really help? ClipToView already assumes that x is sorted and a binary search of 1e7 elements would only take about 23 iterations? The isfinite bypass seems like a straightforward improvement. |
Beta Was this translation helpful? Give feedback.
-
|
Yes, a lot. Here I compare with and without With Without I attach a screen capture of how fluid it now is to navigate is such a big dataset with those modifications : Capture.video.du.05-05-2023.14.39.06.webmNote that I also added a method to PlotDataset() to inherit properties from another one and avoid to have to repeat it all the time 👍 and at the end of |
Beta Was this translation helpful? Give feedback.
-
|
The poor performance seems to be because In [9]: xi32 = np.arange(1e7, dtype=np.int32)
In [10]: xf32 = np.arange(1e7, dtype=np.float32)
In [11]: xf64 = np.arange(1e7, dtype=np.float64)
In [12]: %timeit np.searchsorted(xi32, 100.0)
33.4 ms ± 1.12 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
In [13]: %timeit np.searchsorted(xf32, 100.0)
32.7 ms ± 1.72 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
In [14]: %timeit np.searchsorted(xf64, 100.0)
2.57 µs ± 36.7 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)I suspect that if you were to replace Or pass in x manually with dtype float64. import pyqtgraph as pg
import numpy as np
pg.mkQApp()
pw = pg.PlotWidget()
pw.setClipToView(True)
pw.enableAutoRange(False)
pw.setDownsampling(ds=None, auto=True, mode="subsample")
data = np.arange(10000000)+np.random.normal(0,10,10000000)
x = np.arange(data.size, dtype=data.dtype)
pw.plot(x, data)
pw.show()
pg.exec() |
Beta Was this translation helpful? Give feedback.
-
|
Indeed !! Wonderful !!! But takes useless memory :-D |
Beta Was this translation helpful? Give feedback.
-
|
We could replace In [3]: xi32 = np.arange(1e7, dtype=np.int32)
In [4]: xf32 = np.arange(1e7, dtype=np.float32)
In [5]: xf64 = np.arange(1e7, dtype=np.float64)
In [7]: %timeit np.searchsorted(xi32, 100.0)
35.7 ms ± 5.28 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
In [8]: %timeit np.searchsorted(xf32, 100.0)
36.1 ms ± 5.49 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
In [9]: %timeit np.searchsorted(xf64, 100.0)
2.71 µs ± 123 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)
In [10]: %timeit bisect.bisect_left(xi32, 100.0)
60 µs ± 4.93 µs per loop (mean ± std. dev. of 7 runs, 10,000 loops each)
In [11]: %timeit bisect.bisect_left(xf32, 100.0)
53.8 µs ± 1.76 µs per loop (mean ± std. dev. of 7 runs, 10,000 loops each)
In [12]: %timeit bisect.bisect_left(xf64, 100.0)
2.42 µs ± 114 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each) |
Beta Was this translation helpful? Give feedback.
-
|
I have created #2719 implementing 2 of the ideas in this discussion. Would appreciate it if you could test it out. |
Beta Was this translation helpful? Give feedback.
-
|
Regarding point 2 of your comment #2719 (comment), would it not be sufficient for you to import pyqtgraph as pg
import numpy as np
pg.mkQApp()
pw = pg.PlotWidget()
pw.show()
pw.setClipToView(True)
pw.enableAutoRange(False)
pw.setDownsampling(ds=None, auto=True, mode="subsample")
data = np.arange(10000000)+np.random.normal(0,10,10000000)
pdi = pw.plot()
pdi.setData(data)
pg.exec()A where it was previously |
Beta Was this translation helpful? Give feedback.
-
|
Yes it is the perfect solution. Also solves point number 1.
Could do the closest power of two: But again with all those solutions it is possible that for the right parameter it might oscillate (but much less often than when all integers are allowed). We could also just break the loop. I think the loop is due to the fact that the max values of the displayed dataset change a little with ds. One solution would be to always add the last point to the downsampled dataset, however I think this is no more basic slicing, and will thus create a copy instead of a view. |
Beta Was this translation helpful? Give feedback.
-
|
Hi everyone. Some of this discussion seem to resemble what I was trying in PR #1574. I thought I'd mention it in case you haven't seen it. The approach there seemed to work pretty well, but I never got around to finishing the PR. |
Beta Was this translation helpful? Give feedback.
-
|
Hi Lidstrom. Indeed very similar question ! Note that PR 2719 by @pijyoi + its hint above (first create the plot, then Try to pull the last dev version, load 10⁸ or 10⁹ points dataset this way and navigate using pan and zoom and you will see. Now very fluid. |
Beta Was this translation helpful? Give feedback.
-
|
Something like the following would break the oscillation: if not hasattr(self, '_last_ds_value'):
self._last_ds_value = ds
if abs(self._last_ds_value - ds) / ds < 0.05:
ds = self._last_ds_value
self._last_ds_value = ds
print(ds, view_range)The stored state would of course need to be reset at appropriate moments. |
Beta Was this translation helpful? Give feedback.
-
Hey !
I'm using pyqtgraph to display very large datasets and I find the autodownsample and cliptoview very useful. However, for larger datasets (>10⁷ pts) it is very slow and I found simple fixes which increase tremendously the performances, bypassing
np.isfiniteandnp.searchsortedwhen they are not necessary. After those little fixes I very smoothly navigate in huge timeseries.This is how I increase a lot the performance in my fork :
About downsampling
About ClipToView
[EDIT : as @pijyoi noted the poor performance of
np.searchsortedwas due to the fact that my xArray was of typeint. Usingnp.searchsortedwith aintarray while the second argument is afloatis extremely unefficient, hence my problem. So just make sure your xarray has a typenp.float64. This should also be corrected in the code of pyqtgraph, replacingnp.arange(len(y))bynp.arange(len(y),dtype=np.float64)in the initialization ofPlotDataItemwhen no x array is passed].If the xArray is sorted and linear (proportional to indices), I do not use
np.searchsortedanymore. To do that I added a propertyxArrayLinSortedto the classPlotDataset. It can be set toTrueeither by the user by passing the argumentxArrayLinSorted=Trueto the initlization of thePlotDataItemor is automatically be set in the case of the absence of xArray passed by the user (when initialization of a PlotDataItem directly usenp.arange)I also added the properties
xOriginandxIncrementwhich contains respectivelyx[0]andx[1]-x[0]PlotDataItem update problem bypass
Also there is a problem when you start a plot. The PlotDataItem gets updated before the autodownsample or downsample properties get passed to it by the
PlotItem. To avoid that I must repeat the argumentsautoDownsample=1,downsample=None,downsampleMethod="subsample"in theplotfunction even if those properties are set in thePlotItem.Not Fixed : many calls to
_getDisplayDatasetIn general when you start a plot,
_getDislayDatasetis called around 5 times because of many functions calling updateItems. I do not think it is necessary and maybe you can find a workaround to increase performance.Further not tested ideas
Downsambling with "subsample" does not take much time but for the others method you might want to avoid to repeat that operation at each update, for example by saving resampled arrays somewhere (in _datasetMapped ?) for later reuse
If the autosampling mode is off and the cliptoview mode is on, only compute the resampled array once and apply cliptoview on the resampled array. Do not resample because of cliptoview.
Note that the condition above which forces the update has a redundancy. It it is also forced by the self._dataDisplay = None in viewRangeChanged
Beta Was this translation helpful? Give feedback.
All reactions