Arrow support in future plans?

[mkcorneli]

Thinking about the reader architecture: have you considered an Arrow-native path via ADBC alongside the existing ODBC reader?

Reasoning: Polars is already the internal DataFrame format, so an AdbcReader could hand Arrow record batches to Polars zero-copy, avoiding the row-by-row accumulation in OdbcReader's ColumnBuilder. Would complement rather than replace ODBC since ADBC driver coverage is narrower, but for backends that ship ADBC drivers it's a cleaner path.

This builds off another feature request (see [330] (#330) ) .Exasol ships exarrow-rs, an ADBC-compatible Rust driver, on crates.io. So an AdbcReader would unlock Exasol immediately without needing a dialect-specific module. Other warehouses are on the same trajectory (Snowflake, DuckDB, Postgres all have ADBC drivers shipping). Either way I'd love to contribute, but want to align with your team's technical direction!

[thomasp85]

ADBC is certainly something we should be thinking about, so we are aligned there. I'm actually in the process of attempting to remove polars since it's a horribly heavy dependency for what we use it for, and instead rely on arrow for our internal data frame representation. I'm not sure how that work factors into your proposal.

I won't stop you from making any PRs at the moment, but, we are still in early days and have a lot of work that might come before it so don't be dissuaded if it is not merged in immediately.

If an ADBC reader is your main concern then I can pretty much guarantee that we'll be working on it. Snowflake is a prime use case in our minds and anything we can do to make that connection as efficient as possible will be pursued

[evanlhatch]

@thomasp85

Given you're trying to remove Polars have you looked at Apache Datafusion?

It is better known for SQL but has a dataframe API too. It uses arrow as in-memory format ofc.

Vegafusion also uses it as a core dep so already prior art for using with vegalite.

Would love if there was explcit support for it as with duckdb

[thomasp85]

For the moment my plan is to rely on base arrow. Datafusion is certainly interesting as a reader but if I can get by slimming down the dependencies of the Plot module to the bare minimum I'd like to do that