fix(lineage) Fix lineage viz with multiple siblings

[chriscollins3456]

Fixes a bug we were seeing in the lineage visualization where there were multiple sibling nodes. The bug was that we weren't getting all of the sibling lineages for non-central nodes so if the node with multiple siblings wasn't the sibling node, we would only get their lineage from the urn we have, and not any of the lineages of their siblings. So it wouldn't show non-central node siblings lineage at all, and things would look like the explicit lineage described by individual aspects, not the combined siblings lineage. If we the center node is the one with multiple siblings it worked just fine.

This fixes it by calling the sibling graph service getLineage method for all siblings as well as the center, original request node (as opposed to the regular graph service that just gets lineage from the graph based on the singular urn provided). The siblings graph service aggregates all sibling lineage for a given node to get all of its siblings lineage as well. I need to keep track of the visited urns so that we don't get stuck in an infinite loop of siblings and fetching.

Checklist

• The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
• Links to related issues (if applicable)
• Tests for the changes have been added/updated (if applicable)
• Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
• For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

[jjoyce0510]

Should we have any test cases for this? In SiblingGraphServiceTest

[chriscollins3456]

sure thing - I'll add that now! just wanted to get the initial change up for y'all to look at first

[jjoyce0510]

TBH I'm still a bit confused about what was happening here. But the code changes make sense.

So basically if we get into a state where there are 3 siblings, and sibling 1 points to sibling 2, but sibling 2 points to sibling 1 AND sibling 3, then we will not be able to resolve the full sibling set?

I guess a followup question would be: how is it that we can even arrive at this state in the first place? Shouldn't the sibling hook handle this type of synchronization?

[chriscollins3456]

So basically if we get into a state where there are 3 siblings, and sibling 1 points to sibling 2, but sibling 2 points to sibling 1 AND sibling 3, then we will not be able to resolve the full sibling set?

@jjoyce0510 yeah exactly - if we're getting lineage for sibling 1, then we get sibling 2's lineage but not sibling 3's lineage. that's because in this resolver we get all of sibling 1's siblings and their lineage and aggregate, but we don't get any of the siblings of sibling 1 (aka sibling 3) because we simply get the lineage for sibling 2 by hitting the regular graph service. Now, we can get all of sibling 2's siblings and their lineage as well (thus getting sibling 3 and its lineage).

I guess a followup question would be: how is it that we can even arrive at this state in the first place? Shouldn't the sibling hook handle this type of synchronization?

This situation can happen when we're fetching data for a sibling that doesn't have the full sibling set explicitly set on it (ie. fetching data for sibling 1 and not sibling 2, as sibling 2 points to 1 and 3). We always fetch data for all the siblings we explicitly define, but when there are siblings of siblings we aren't getting those. On the entity page it isn't as big of a deal because we explicitly show you who the siblings are for the page you're on (here this is the main sibling with all siblings attached):

while this is one of the other combined siblings that is a sibling of the main one as well but doesn't have a siblings set to everything that its sibling has siblings for:

You can navigate to get to the central sibling clicking on those entity cards on the bottom right.

I think the biggest issue here is when lineage looks wrong because we're not combining siblings there correctly at all unless you have the main sibling (that has all of the sibling relationships defined on it) selected as the center node. and then it will look correct only for that grouping of siblings, while upstream or downstream groupings may have been wrong (but look good with this change)

[jjoyce0510]

Thanks for the great explanation! It's much clearer to me now... Once build and test passing let's get this in!

[jjoyce0510]

LGTM. Awesome work!