You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Why holders.get_column_lineage return simple_paths which equal to one only? Are there any reasons?
In my case, i have a target column (i.e. with alias X) which is the result of a CASE construction, and all of those columns (A,B,C) are from same column and same table (i.e. BASE_TABLE.Z), so result of nx.all_simple_paths returns the next paths:
BASE_TABLE.Z->A->X
BASE_TABLE.Z->B->X
BASE_TABLE.Z->C->X
But get_column_lineage not return any one, the reason is the following lines:
if len(simple_paths) == 1:
columns.add(tuple(simple_paths[0]))
(simplified example)
CASE
when JOIN_TABLE_1.A not null JOIN_TABLE_1.A
when JOIN_TABLE_2.B not null JOIN_TABLE_2.B
else JOIN_TABLE_3.C
END as X
JOIN SELECT Z as A FROM BASE_TABLE as JOIN_TABLE_1
JOIN SELECT Z as B FROM BASE_TABLE as JOIN_TABLE_2
JOIN SELECT Z as C FROM BASE_TABLE as JOIN_TABLE_3
Thank you in advance!
The text was updated successfully, but these errors were encountered:
Thanks for reporting this. This is a bug. Actually if you see the comment in the code, you'll find that I didn't know back then whether there would be more than one simple path. And now you give a valid example. We should get this fixed.
INSERT OVERWRTIE TABLE foo
SELECT
CASE
when JOIN_TABLE_1.Anot nullJOIN_TABLE_1.A
when JOIN_TABLE_2.Bnot nullJOIN_TABLE_2.B
else JOIN_TABLE_3.C
END as X
FROM JOIN_TABLE_0
JOIN (SELECT Z as A FROM BASE_TABLE) as JOIN_TABLE_1
JOIN (SELECT Z as B FROM BASE_TABLE) as JOIN_TABLE_2
JOIN (SELECT Z as C FROM BASE_TABLE) as JOIN_TABLE_3
reata
changed the title
Why holders.get_column_lineage return simple_paths which equal to one only
Failure to Handle Multiple Lineage Path for Same Column
Mar 13, 2022
Why holders.get_column_lineage return simple_paths which equal to one only? Are there any reasons?
In my case, i have a target column (i.e. with alias X) which is the result of a CASE construction, and all of those columns (A,B,C) are from same column and same table (i.e. BASE_TABLE.Z), so result of nx.all_simple_paths returns the next paths:
But get_column_lineage not return any one, the reason is the following lines:
if len(simple_paths) == 1:
columns.add(tuple(simple_paths[0]))
(simplified example)
CASE
when JOIN_TABLE_1.A not null JOIN_TABLE_1.A
when JOIN_TABLE_2.B not null JOIN_TABLE_2.B
else JOIN_TABLE_3.C
END as X
JOIN SELECT Z as A FROM BASE_TABLE as JOIN_TABLE_1
JOIN SELECT Z as B FROM BASE_TABLE as JOIN_TABLE_2
JOIN SELECT Z as C FROM BASE_TABLE as JOIN_TABLE_3
Thank you in advance!
The text was updated successfully, but these errors were encountered: