-
Notifications
You must be signed in to change notification settings - Fork 1.8k
Closed
Labels
bugSomething isn't workingSomething isn't working
Description
Describe the bug
Recently we made a change in delta-rs to make the schema's agnostic of large/normal/view types. However datafusion can't properly coerce List and LargeList yet
To Reproduce
import polars as pl
tmp_path = "test_table"
pl.DataFrame({"a": ["test"], "b": [[1]]}).write_delta(tmp_path)
pl.DataFrame({"a": ["test1"], "b": [[1]]}).write_delta(
tmp_path,
mode="merge",
delta_merge_options={
"predicate": "s.a = t.a",
"source_alias": "s",
"target_alias": "t",
},
).when_not_matched_insert_all().execute()DeltaError: Generic DeltaTable error: type_coercion
caused by
Error during planning: Failed to coerce then ([LargeList(Field { name: "item", data_type: Int64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }), List(Field { name: "item", data_type: Int64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }), List(Field { name: "item", data_type: Int64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }), List(Field { name: "item", data_type: Int64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} })]) and else (None) to common types in CASE WHEN expression
Expected behavior
Be able to coerce any list type combination, whether its large, view or normal
Additional context
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working