Unified ibis <-> pandas type conversions

[kszucs]

I'd like to implement a method to insert pandas dataframes to clickhouse table.

Problem

Actually this implementation is quite straightforward, but I'd like a unified way to convert dtypes between pandas, ibis [and clickhouse].

Currently this sort of type mapping logic has been implemented at multiple places:

• https://github.com/ibis-project/ibis/blob/master/ibis/clickhouse/types.py
• https://github.com/ibis-project/ibis/blob/master/ibis/pandas/client.py#L35
• https://github.com/ibis-project/ibis/blob/master/ibis/impala/pandas_interop.py#L41
• https://github.com/ibis-project/ibis/blob/master/ibis/sql/alchemy.py#L687

Proposal

AFAIK the only supported data container is pd.DataFrame:

• *query.execute() -> DataFrame
• *table.insert(obj: DataFrame)

Ideally there should be two type conversion paths:

1. Insertion or external data see Support external tables for clickhouse #1164

   pandas dtypes -> ibis schema -> backend schema [in this case clickhouse]

2. Selection

   backend schema -> ibis schema -> pandas dtypes

I think the highlighted parts should be placed in a single module which is reachable from every backend package, this way we can factor out the redundant implementations.

What do You think @cpcloud @wesm ?

[wesm]

This makes sense. This probably also should be considered together with Arrow types, which need to have a reasonable mapping to Ibis types. cc @jreback

[jreback]

#1194
maps arrow to ibis types ; could/should be in the same place

[kszucs]

@jreback what's the status of that PR?

If You agree with my proposal, where should we place that type conversion logic?

• go to ibis.pandas_interop
• or just make multipledispatch mandatory and implement in ibis.pandas?

Should I go with it, or perhaps You'll implement it?

[jreback]

@kszucs that will merge soon. yes I think we should make a ibis.types where we can move all types of type conversion code

[jreback]

yeah I think we should just make multipledispatch as required, as pandas already is

[kszucs]

@jreback Great! Would You mind creating the skeleton of it?

I could help then to implement the ibis schema <-> backend schema conversions per backend (e.g. to ibis.clickhouse.types)

Should we include infer_literal_types from expr package too?

[kszucs]

@cpcloud could You give me a proper mapping for implementingto_pandas ?
A preliminary attempt was here 93bd201

[cpcloud]

@kszucs I think if you just glom all of those types into a dict somewhere then you have most of what you need. Let me know if there's something that isn't clear.

[kszucs]

Ok!

[kszucs]

This should be resolved by #1292

[kszucs]

@cpcloud I think You can close this.