Skip to content

LargeList and List type coercion not working in CASE WHEN #12370

@ion-elgreco

Description

@ion-elgreco

Describe the bug

Recently we made a change in delta-rs to make the schema's agnostic of large/normal/view types. However datafusion can't properly coerce List and LargeList yet

To Reproduce

import polars as pl
tmp_path = "test_table"

pl.DataFrame({"a": ["test"], "b": [[1]]}).write_delta(tmp_path)
pl.DataFrame({"a": ["test1"], "b": [[1]]}).write_delta(
    tmp_path,
    mode="merge",
    delta_merge_options={
        "predicate": "s.a = t.a",
        "source_alias": "s",
        "target_alias": "t",
    },
).when_not_matched_insert_all().execute()
DeltaError: Generic DeltaTable error: type_coercion
caused by
Error during planning: Failed to coerce then ([LargeList(Field { name: "item", data_type: Int64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }), List(Field { name: "item", data_type: Int64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }), List(Field { name: "item", data_type: Int64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }), List(Field { name: "item", data_type: Int64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} })]) and else (None) to common types in CASE WHEN expression

Expected behavior

Be able to coerce any list type combination, whether its large, view or normal

Additional context

delta-io/delta-rs#2851

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions