Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

'common_sub_expression_eliminate' failed on unnested structs field #10990

Closed
ahirner opened this issue Jun 19, 2024 · 2 comments · Fixed by #10991
Closed

'common_sub_expression_eliminate' failed on unnested structs field #10990

ahirner opened this issue Jun 19, 2024 · 2 comments · Fixed by #10991
Labels
bug Something isn't working

Comments

@ahirner
Copy link

ahirner commented Jun 19, 2024

Describe the bug

After filter pushdown in #10935 the optimizer does seem misunderstand struct fields.

To Reproduce

Create a table with a list of structs:

CREATE TABLE d AS VALUES(1,[named_struct('a', 1, 'b', 2)]),(2,[named_struct('a', 3, 'b', 4), named_struct('a', 5, 'b', 6)]);

Observe failure:

> select * from (select column1, unnest(column2) as o from d) where o['a'] = 1;
Optimizer rule 'common_sub_expression_eliminate' failed
caused by
Error during planning: The expression to get an indexed field is only valid for `List`, `Struct`, `Map` or `Null` types, got List(Field { name: "item", data_type: Struct([Field { name: "a", data_type: Int64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "b", data_type: Int64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} })

Expected behavior

> select * from (select column1, unnest(column2) as o from d) where o['a'] = 1;
+---------+--------------+
| column1 | o            |
+---------+--------------+
| 1       | {a: 1, b: 2} |
+---------+--------------+

Additional context

Maybe duplicate of #10989

@ahirner ahirner added the bug Something isn't working label Jun 19, 2024
@jayzhan211
Copy link
Contributor

jayzhan211 commented Jun 19, 2024

I believe it is the same issue of #10989 , but we can keep this opened until it is solved

@alamb
Copy link
Contributor

alamb commented Jun 19, 2024

I verified this test case works after #10991

sql
     Running `target/debug/datafusion-cli`
DataFusion CLI v39.0.0
> CREATE TABLE d AS VALUES(1,[named_struct('a', 1, 'b', 2)]),(2,[named_struct('a', 3, 'b', 4), named_struct('a', 5, 'b', 6)]);

0 row(s) fetched.
Elapsed 0.031 seconds.

> select * from (select column1, unnest(column2) as o from d) where o['a'] = 1;
+---------+--------------+
| column1 | o            |
+---------+--------------+
| 1       | {a: 1, b: 2} |
+---------+--------------+
1 row(s) fetched.
Elapsed 0.025 seconds.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants