feat(pyarrow): support __arrow_c_schema__ on ibis.Schema objects
#9665
Conversation
| pytest.importorskip("pyarrow") | ||
| schema = sch.Schema({"a": dt.int64, "b": dt.string, "c": dt.boolean}) | ||
| # smoketest, since no way to create schema from capsule in current pyarrow | ||
| assert schema.__arrow_c_schema__() is not None |
There was a problem hiding this comment.
pa.schema will use the __arrow_c_schema__ method for recent pyarrow (v15+ I think).
I sometimes use a wrapper class in tests to ensure it sees only the __arrow_c_schema__ method:
class CSchemaWrapper:
def __init__(obj):
self.obj = obj
def __arrow_c_schema__(self):
return self.obj.__arrow_c_schema__()then check
pa.schema(CSchemaWrapper(schema))
There was a problem hiding this comment.
fwiw, I tried this (pa.schema(ibis_schema)) with the ibis 9.2.0 and pyarrow 17.0 and alas, no dice:
File [~/test-hook/lib/python3.11/site-packages/pyarrow/types.pxi:5360](https://notebooks.freenome.net/user/mcorley/lab/tree/test-hook/lib/python3.11/site-packages/pyarrow/types.pxi#line=5359), in pyarrow.lib.schema()
File [~/test-hook/lib/python3.11/site-packages/pyarrow/types.pxi:3541](https://notebooks.freenome.net/user/mcorley/lab/tree/test-hook/lib/python3.11/site-packages/pyarrow/types.pxi#line=3540), in pyarrow.lib.field()
File [~/test-hook/lib/python3.11/site-packages/pyarrow/types.pxi:5301](https://notebooks.freenome.net/user/mcorley/lab/tree/test-hook/lib/python3.11/site-packages/pyarrow/types.pxi#line=5300), in pyarrow.lib.ensure_type()
TypeError: DataType expected, got <class 'ibis.expr.datatypes.core.String'>
There was a problem hiding this comment.
There's something odd going on with our base classes and whatever mechanism that pyarrow is using to invoke the method.
There was a problem hiding this comment.
Arg, pyarrow first checks whether the input is a Mapping and then checks whether __arrow_c_schema__ exists 😞
There was a problem hiding this comment.
Seems like this functionality is broken in pyarrow: an object can't be both a mapping and implement the __arrow_c_schema__ interface.
There was a problem hiding this comment.
Opened up apache/arrow#43388 to track this.
There was a problem hiding this comment.
In the meantime I'll add an xfailing test that will start to pass once this is addressed.
Part of #9660.