-
Notifications
You must be signed in to change notification settings - Fork 746
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
GOBBLIN-616: Add ability to fork jobs when concatenating Dags. #2483
Conversation
otherNode.addParentNode(node); | ||
} | ||
} else { | ||
for (DagNode otherNode: other.startNodes) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can a parent of a forkable node also be forkable?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No. Only nodes which are in the list of "endNodes" of a Dag are eligible for being forkable.
//Append all the entries from the other dag's parentChildMap to this dag's parentChildMap | ||
for (Map.Entry<DagNode, List<DagNode<T>>> entry: other.parentChildMap.entrySet()) { | ||
for (Iterator<Map.Entry<DagNode, List<DagNode<T>>>> iterator = other.parentChildMap.entrySet().iterator(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why not use Map.putAll()?
Addressed the requested change. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1
Closes apache#2483 from sv2000/dependents
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
JIRA
Description
We add the ability to fork jobs when concatenating 2 dags, as part of constructing a multi-hop path. An example use case is to allow retention jobs on each hop of the path to run in parallel to the distcp jobs along a multi-hop path, thus preventing the data copy on a subsequent hop from being blocked on a retention job on a previous hop.
Tests
Added test cases to DagTest and FlowGraphPathTest classes.
Commits