Support array for arrow

[wonb168]

array column like: {1,2,3}
df=cx.read_sql(con_pg, sql,return_type='arrow2')
RuntimeError: No conversion rule from Int4Array(true) to connectorx::destinations::arrow::typesystem::ArrowTypeSystem.

[wangxiaoying]

Hi @wonb168 , thanks for opening the issue. Unfortunately, we currently do not support list type in arrow. Will add this to our todo list.

[wonb168]

Hi @wonb168 , thanks for opening the issue. Unfortunately, we currently do not support list type in arrow. Will add this to our todo list.

but why I use pandas's read_sql, and then change to arrow is OK?
Unfortunately, pandas changed int to float if have null in int column, and pandas is slow.

[wangxiaoying]

Hi @wonb168 , thanks for opening the issue. Unfortunately, we currently do not support list type in arrow. Will add this to our todo list.

but why I use pandas's read_sql, and then change to arrow is OK? Unfortunately, pandas changed int to float if have null in int column, and pandas is slow.

If the destination is set to arrow, connectorx will directly write the result into arrow instead of writing to pandas first and convert to arrow. Currently array type is supported for pandas in connectorx but not for arrow/arrow2.

[wangxiaoying]

Hi @wonb168 , we have added array support in the newest alpha version for arrow2. Please feel free to try pip install connectorx==0.3.1a7 and test whether it works.