bug: KeyError: JsonType(extension<arrow.json>)

[timoha]

What happened?

When reading a DeltaTable with DataFusion backend, getting the following exception:

  File "/Users/andrey/src/tessera/.venv/lib/python3.13/site-packages/ibis/backends/datafusion/__init__.py", line 408, in get_schema
    return sch.schema(table.schema)
           ~~~~~~~~~~^^^^^^^^^^^^^^
  File "/Users/andrey/src/tessera/.venv/lib/python3.13/site-packages/ibis/common/dispatch.py", line 115, in call
    return impl(arg, *args, **kwargs)
  File "/Users/andrey/src/tessera/.venv/lib/python3.13/site-packages/ibis/expr/schema.py", line 357, in from_pyarrow_schema
    return PyArrowSchema.to_ibis(schema)
           ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
  File "/Users/andrey/src/tessera/.venv/lib/python3.13/site-packages/ibis/formats/pyarrow.py", line 265, in to_ibis
    fields = [(f.name, PyArrowType.to_ibis(f.type, f.nullable)) for f in schema]
                       ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File "/Users/andrey/src/tessera/.venv/lib/python3.13/site-packages/ibis/formats/pyarrow.py", line 160, in to_ibis
    return _from_pyarrow_types[typ](nullable=nullable)
           ~~~~~~~~~~~~~~~~~~~^^^^^
KeyError: JsonType(extension<arrow.json>)

The schema of the delta table is as follows, specifically pa.json_() type for column response is used:

_RESPONSE_SCHEMA = pa.schema({
    "archive": pa.string(),
    "document": pa.string(),
    "page": pa.int64(),
    "as_of": pa.timestamp("s"),
    "response": pa.json_(),
    "response_id": pa.string(),
    "run_id": pa.string(),
})

Upon inspection of ibis code looks like it doesn't support arrow.json and expects a string from JSON type.

What version of ibis are you using?

ibis-framework[datafusion] = 10.6.0

What backend(s) are you using, if any?

datafusion = 47.0.0

Relevant log output

Code of Conduct

• I agree to follow this project's Code of Conduct

[timoha]

My current workaround:

from ibis.formats import pyarrow as ibis_pa
import ibis.expr.datatypes as dt


ibis_pa._from_pyarrow_types[pa.json_()] = dt.JSON

Let me know if you'd like me to submit a PR just adding the entry + tests

[cpcloud]

Thanks for the report, would love a PR for this!

[cpcloud]

Hm, this might be trickier than it looks since we need to keep backwards compatibility with older versions of arrow that may not have JsonType.

[timoha]

Just so that I understand, your concern is when a write/convert to arrow is done, what type should be used to return in arrow? So, if we blindly decide to convert to arrow json, it can impact a lot of existing code that expects it marshaled as string?

And on the read side, it would now start recognizing arrow json type and attributing it to correct ibis type whereas it'd fail before. So there isn't really a problem with backwards compatibility on the read side.

[cpcloud]

Read side is probably not that risky, but there's still a potential issue with backends that directly operate on arrow tables, like duckdb, datafusion and polars. They would have to be aware of the Json type. Realistically it's probably not a huge deal though since all those projects are actively developed.