Join Statement: Schema contains duplicate unqualified field name

[alamb]

Note: migrated from original JIRA: https://issues.apache.org/jira/browse/ARROW-11432

apache/arrow#9307

[alamb]

Comment from GANG LIAO(gangliao) @ 2021-01-29T20:09:31.033+0000:

join statement cannot distinguish two columns with the same name from two tables. [https://github.com/apache/arrow/issues/9307]

[~jorgecarleitao]  

Comment from R J(TurnOfACard) @ 2021-02-18T11:19:34.301+0000:

When building a schema using `datafusion::logical_plan::plan::LogicalPlan::schema()` the returned schema lacks a table qualifier. I think when building the TableScan, it allocates a table_name, but the returned`datafusion::logical_plan::dfschema::DFField`'s have no reference to the provided table name. Attempting to join the schemas (in `datafusion::logical_plan::dfschema::DFSchema::new`) then results in a list of fields with conflicting names as they lack qualifiers.

I am new to DataFusion but would be happy to add some unit tests and try to look at this on the weekend.

Comment from R J(TurnOfACard) @ 2021-02-20T22:59:14.507+0000:

I'm not sure there is an easy fix without making breaking changes to the public API. When building a join schema, it checks if the join set is valid (physical_plan::hash_utils::check_join_set_is_valid), which has a parent public API call (physical_plan::hash_utils::check_join_is_valid). This join is unaware of the registered name (CSV or parquet) as it is performed with arrow schemas rather than DataFusion schemas.

 

EDIT:

It could be my lack of knowledge of the DataFusion codebase, but it appears it would need a lot of changes.