Processing speed with large datasets

[jmakov]

How exactly does perspective support large data e.g. +10M points. My experience is waiting about 1 min to e.g. process perspective.Table(polars_df.to_pandas()). The same with view. And the widget was then unusable. Or perhaps I'm not using it correctly? With e.g. datashader I can just throw my multi gig data frame in it and see a plot 2-5sec later. If anyone can suggest how perspective is intended to be used in this kind of context, please let me know.

[sinistersnare]

If you can provide some code that reproduces this, especially wrt a comparison to datashader, this would be a very valuable bug report! Thanks.

[jmakov]

Perhaps I'd need a smaller data set that would be updated as we navigate through the data somehow. Thought pandas.LazyFrame and arrow streaming would automagically be used (when using polars.scan_pyarrow_dataset). I assume I'm doing something very wrong here.

from datetime import date, datetime

import numpy
import polars
import perspective

# works great with 10_000
ENTRIES = 10_000_000

df = polars.DataFrame({
    "col1_int": numpy.arange(ENTRIES),
    "col2_int": numpy.arange(ENTRIES),
})

# this takes +10s
table = perspective.Table(df.to_pandas())
# this takes +10s
view = table.view()
# this takes +10s
df_view = view.to_df()
# another +10s
df_view = view.to_df()

# unusable (no data displayed)
perspective.PerspectiveWidget(df_view)

# datashader version
import hvplot
import hvplot.pandas  # noqa

df.to_pandas().hvplot(rasterize=True)

[timkpaine]

For more details:

df = polars.DataFrame({
"col1_int": numpy.arange(ENTRIES),
"col2_int": numpy.arange(ENTRIES),
})

You make a polars dataframe here. This will give you an arrow table underneath

table = perspective.Table(df.to_pandas())

You start by going slower here. Perspective can natively read arrow, but you wrap it in a pandas dataframe which must be processed from python. Then you ingest this pandas table into perspective, which is a (justifiably) slower operation than e.g. just rendering a view on the dataframe, since perspective is a whole data engine and not just a visualization component, so it will copy the data from the dataframe into the engine. This is much faster via arrow, as @texodus's answer describes.

df_view = view.to_df()

no reason to take a view and go back to a dataframe when you had a dataframe to begin with, and again an arrow is orders of magnitude faster as it can be done natively.

df_view = view.to_df()

same

perspective.PerspectiveWidget(df_view)

explained below

df.to_pandas().hvplot(rasterize=True)

reading off an existing datastore to make a plot will always be "faster" than creating a new data engine, copying the data into it, and then rendering a plot, at least for initial load time.

[texodus]

@jmakov The widget is unusable because you're using it incorrectly (and it prints an error to this effect PerspectiveError: Only 'Table' or data can be loaded, so I'm not sure why you're asking about it here):

# Don't pass a `View` here
perspective.PerspectiveWidget(table)

It's slow because you are doing processing gymnastics on it before loading it. Use an Apache Arrow if you want the fastest loading time possible, not a DataFrame:

After the table is loaded, the performance differences evaporate:

[jmakov]

Thanks for the quick feedback. It is a bit strange though that somebody else than the OP decides when the question was answered and which answer was the solution...

[timkpaine]

usually they don't bother to pick an answer, or they pick a sub-par one which makes it harder for future people with the same question ¯\(ツ)/¯