vega-lite-api & nodejs-polars

[BorisTherin]

Hi awesome team.

I'm working on a jupyterlab notebooks project with nodesjs-polars (with deno kernel), and sounds like vega-lite-api does not support polars dataframe.
Unfortunatly, nodejs-polars does not provide the to_pandas() method (like the python version).

As far as feature parity goes. Ideally i'd like to write some scripts that generate a coverage map between node & python features. We know some features will be much more difficult to support in JS (if not impossible). (lambdas, and interop methods [pandas, arrow]). But until we have some automated way of tracking this, it is a manual task to update the features to match py-polars.
from pola-rs/nodejs-polars#30

so atm i have to pass my dataframe to vega-ite-api with nodejs-polars writeCSV() method this way.

const response = await fetch("https://raw.githubusercontent.com/vega/vega/main/docs/data/gapminder-health-income.csv")
const data = await response.text()
const df = pl.readCSV(data)

/*
    our every polars manipulations
*/

const plot = vl
    .markCircle()
    .params(vl.selectInterval().bind("scales"))
    .width(700)
    .data(
        df.writeCSV()   // convert to CVS buffer
    )
    .encode(
        vl.x().field("income").scale({ type: "log"}),
        vl.y().fieldQ("health").scale({zero: false}).axis({ minExtent: 30 }),
        vl.size(vl.fieldQ("population")),
        vl.color({ value: "#262696" }),
    );
display(plot) 

if i try to pass polars original dataframe .data(df) it do crash my deno kernel every time

is polars (actually nodes-polars) dataframe aimed to be suport by vega-lite-api in the project's roadmap ?
or am i doing something wrong by my side ?

[domoritz]

Vega-Lite supports arrow so could you export the underlying arrow from polars?

[BorisTherin]

from nodejs-polars issue vega/vega-lite-api#30
We know some features will be much more difficult to support in JS (if not impossible). (lambdas, and interop methods [pandas, arrow]).
pandas & arrow are not supported in the nodejs version atm, and the (if not impossible) comment doesnt sounds very good here

when i seach for to_XXX_ method in nodejs-polars documentation:

So, having to convert my polar dataframe back into cvs with pl.writeCSV() is not that dramatic, but i will have to defend vega-lite-api use in this project to my team, and knowing if it will be supported would help. polars (actually nodejs-polars) will not be negociable as we will need very large dataset to work with, & polars is the only one providing dataframe larger than our ram.

[jheer]

It looks like this concerns Vega-Lite or really Vega. If Vega can support Polars then it will be easy to update this API to pass that through, just like other data formats.

@domoritz should we transfer out of vega-lite-api for better tracking?

[BorisTherin]

@jheer yes, as polars python version does suppot to_pandas everyhting was fine, but sounds like the javascript side as been forgotten when nodejs-polars cant provide such a function

[domoritz]

Moved to Vega

[BorisTherin]

thx for help and very quick response