feat: Lightweight table wrapper around existing dataframes (polars, pandas)

[vspinu]

What is the motivation behind your request?

I write data manipulation backend agnostic functions which take a table input and do the computation. The problem is that there is no lightweight way to wrap an existing object (ex. polars) into ibis.Table object.

Currently the only way seems to be the memtable which has two "issues":

• it relies on the global backend declaration and
• seems to be performing coercions (data duplication) under the hood even when the backend is the same (I am not completely sure about this one)

import polars as pl
import ibis

df = pl.DataFrame({
    "column1": [1, 2, 3],
    "column2": ["a", "b", "c"]
})

ibis.memtable(df).get_backend()
##  <ibis.backends.duckdb.Backend at 0x13e7ece90>


ibis.set_backend("polars")
tt = ibis.memtable(df)
df is tt.to_polars()
## False

Describe the solution you'd like

I would like to have a lightweight ibis.wrap function which would take an object of the supported backend (ex. polars, pyarrow etc) and automatically wrap it into the corresponding backend in the most lightweight way possible, ideally without any conversions by keeping track of the reference of the original object.

What version of ibis are you running?

10.5.0

What backend(s) are you using, if any?

polars 1.27.1

Code of Conduct

• I agree to follow this project's Code of Conduct

[cpcloud]

Thanks for the issue.

it relies on the global backend declaration

It doesn't. You can use any memtable with a direct con.execute. You cannot use the execute method on a memtable without a global backend, because we have no idea where you want to execute that memtable without a global backend set.

seems to be performing coercion

Do you have a code path or an example that is making you suspect this is happening?

[vspinu]

It doesn't. You can use any memtable with a direct con.execute.

I see. I was not aware of this. So this would execute regardless of the default backend:

con = ibis.polars.connect()
tbl = ibis.memtable(df)
con.execute(tbl.filter(...).count())

You cannot use the execute method on a memtable without a global backend, because we have no idea where you want to execute that memtable without a global backend set.

I understand this is the current implementation but is there a downside in making it global state agnostic? Intuitively it seems reasonable to be able to use ibis syntax on any supported object without relying on global state.

Essentially, I want to be able instead of the 3 lines above simply do ibis.wrap(df).filter(...).count() on the fly. The ibis.wrap would infer the backend from the object type and store it as part of the Table object. Just like what con.table does with registered tables.

Alternatively, an even better interface IMO, would be to use con.table directly on the object to automatically register it within the database. So instead of the current:

con = ibis.polars.connect({"df": df})
tbl = con.table("df")

simply do on the fly:

con = ibis.polars.connect()
tbl = con.table(df)

Do you have a code path or an example that is making you suspect this is happening?

I don't know enough about the pycapsule to be able to get to the internal memory layouts in order to infer if it's exactly the same underlying object. But I see that the internally tracked object is exactly the same one I pass, but tbl.to_polars() does run the "coerssion" loop to_arrow() and then to_polars():

Here is the code:

tbl = ibis.memtable(df)
df is tbl.to_polars()  # False
df is tbl._arg.data.to_polars(None) # True
df is tbl._arg.data.obj # True

Would it be possible to clarify in the docs of memtable what's exactly happening in terms of the data copy and how this interacts with the default backend?

Particularly, is it true that regardless of the backend (arrow, duckdb or polars), as long as the backend understands pyarrow in memory format backend will operate on the original object passed to the memtable?

[vspinu]

I see. I was not aware of this. So this would execute regardless of the default backend:

I have just tried this and, confusingly, execute returns pandas instead of polars 🤔

import polars as pl
import ibis

df = pl.DataFrame({
    "column1": [1, 2, 3],
    "column2": ["a", "b", "c"]
})
plcon = ibis.polars.connect()
ptbl = ibis.memtable(df).filter(_.column1 == 3)
type(plcon.execute(ptbl))
## pandas.core.frame.DataFrame

[gforsyth]

execute is an alias to to_pandas -- results are coerced into a pandas data frame regardless of the execution backend.
You probably want to_polars

[MarcoGorelli]

This might be related to #11208 ?