fix(ui) Fix displaying column level lineage for sibling nodes #7955
Conversation
| @@ -146,3 +146,20 @@ export const handleBatchError = (urns, e, defaultMessage) => { | |||
| } | |||
| return defaultMessage; | |||
| }; | |||
|
|
|||
| // put all of the fineGrainedLineages for a given entity and its siblings into one array so we have all of it in one place | |||
| export function getFineGrainedLineageWithSiblings( | |||
There was a problem hiding this comment.
This is awesome!
|
|
||
| // upstreamEntityUrn could belong to a sibling we don't "render", so store its inputs to updateFineGrainedMap | ||
| // and update the fine grained map later when we see the entity with these siblings | ||
| // eslint-disable-next-line no-param-reassign |
There was a problem hiding this comment.
this is a bit challenging for me to grasp...
There was a problem hiding this comment.
definitely. so when we get fineGrainedLineages for a single entity, we get a combo of an upstream schema field urn and a downstream schema field urn. The downstream schema field always belongs to the current entity at hand. The upstream schema field belongs to some other entity upstream of this one on the graph.
The thing is that this schema field may belong to a sibling of an upstream entity and that sibling doesn't get "rendered" since we collapse sibling nodes. and since all we have is an urn here, we don't know if that upstream urn has siblings or is a sibling itself or what, so I don't know what entity urn I need to pass into updateFineGrainedMap - I need to pass in the urn of the sibling being rendered but I don't know that yet
Also - since fineGrainedLineages belong on downstream entities, the upstream entity doesn't know about it and I can't draw these edges based on its knowledge.
So I have to basically save this upstream urn and its edge so that I can check siblings for every subsequent entity to see if this upstream urn is a sibling, and if so, construct the edge with the correct urn (the urn of the node being rendered, not the sibling urn).
So this solves the case where we have CLL edges for entities we haven't seen yet in the graph. Below, where I loop over fetchedEntities I'm solving for the case where I have already seen the entity (same problem, just one solves for things I haven't seen yet, the other solves for things I have seen already)
I'm sorry this is tough to explain - happy to hop on a call! but also the complexity of (1) this code and (2) our fineGrainedLineage and upstreamLineage makes mapping things real tough. as we're talking about offline, i'm sure there's a better way to fetch and map this data. but that may be smart to solve after this fix solves customer issues.
There was a problem hiding this comment.
Okay got it. Yeah makes sense - seems like siblings + graph is a really tricky combo. As long as this doesn't introduce any regression I'm happy! :)
There was a problem hiding this comment.
haha for sure! things are looking good locally for me but i'll make sure to give it a once more around before merging.
| @@ -161,7 +166,8 @@ export default class EntityRegistry { | |||
| (genericEntityProperties?.upstream?.filtered || 0), | |||
| status: genericEntityProperties?.status, | |||
| siblingPlatforms: genericEntityProperties?.siblingPlatforms, | |||
| fineGrainedLineages: genericEntityProperties?.fineGrainedLineages, | |||
| fineGrainedLineages, | |||
| siblings: genericEntityProperties?.siblings, | |||
There was a problem hiding this comment.
where is this being used?
| @@ -51,6 +52,7 @@ export type FetchedEntity = { | |||
| status?: Maybe<Status>; | |||
| siblingPlatforms?: Maybe<DataPlatform[]>; | |||
| fineGrainedLineages?: [FineGrainedLineage]; | |||
| siblings?: Maybe<SiblingProperties>; | |||
There was a problem hiding this comment.
is this being used on line 83 in constructtree?
There was a problem hiding this comment.
yup it is indeed - we need to the siblings objects so we can get their fineGrainedLineages
We had an issue where we sometimes were not displaying column level lineage properly for nodes that have siblings. This was happening because we had one "main" node that we display and if that node didn't have CLL, then we wouldn't show it. But one of its siblings may have CLL and we should be showing that if so.
This fixes that by creating a mapping of urns to inputs so we can create this
fineGrainedMap(that's used to draw the lines between columns) and we check through each entity's siblings against this new mapping and update thisfineGrainedMapto account for siblings and show CLL on the "visible" sibling node.This ultimately this combines CLL between siblings and shows it as if it were one node.
Checklist