Orchestrators refactoring and fork handling other orchestrators #70
Conversation
|
Progress on this PR is being documented here. |
|
Just refactored orchestrators function to have |
lib/task.js
Outdated
| @@ -39,20 +42,23 @@ const task = (dispatch) => (props, operationCreator, instanceUid = null) => { | |||
| // First thing we need to do is create a `uid` for this task | |||
| // `hashes` is an object with input, output, params keys | |||
| // The `uid` is a hash of the `input`, `output`, and `params` hashes | |||
| const { uid, hashes } = generateUid(props) | |||
| let { uid, hashes } = generateUid(props) | |||
There was a problem hiding this comment.
use const If possible, can use newUid in later part
There was a problem hiding this comment.
This is cool and in fact it is more clear to distinguish between uid and newUid because they are added to different objects. newUid is used when task is in fact created and uid is used out of the scope of the pipeline so to the definition of the task itself before having context.
| @@ -0,0 +1,11 @@ | |||
| 'use strict' | |||
|
|
|||
| const orchestratorUidGenerator = (tasks) => { | |||
There was a problem hiding this comment.
can use .map:
return tasks.map(task => task.info.uid)There was a problem hiding this comment.
This also works well
lib/orchestrators/junction.js
Outdated
| return junctionResults | ||
| }).asCallback(cb) | ||
| } | ||
|
|
||
| junctionInvocator.info = { | ||
| tasks: tasks.map(task => task), // returns an array of tasks |
There was a problem hiding this comment.
can probably do tasks: tasks (unless immutability is important)
There was a problem hiding this comment.
Also seems to work ok.
Codecov Report
@@ Coverage Diff @@
## dev #70 +/- ##
========================================
- Coverage 80.49% 80.2% -0.3%
========================================
Files 36 37 +1
Lines 841 884 +43
Branches 101 104 +3
========================================
+ Hits 677 709 +32
- Misses 164 175 +11
Continue to review full report at Codecov.
|
Codecov Report
@@ Coverage Diff @@
## dev #70 +/- ##
==========================================
+ Coverage 80.49% 81.88% +1.38%
==========================================
Files 36 37 +1
Lines 841 861 +20
Branches 101 102 +1
==========================================
+ Hits 677 705 +28
+ Misses 164 156 -8
Continue to review full report at Codecov.
|
lib/orchestrators/fork.js
Outdated
| return forkResults | ||
| }).asCallback(cb) | ||
| } | ||
| const forkInvocator = (cb = _.noop, ctx = defaultContext) => {} |
This pull request includes a patch on uid for spamming tasks with junction as discussed in #69. Though this is still incomplete the remaining code of the PR can be quite complex and so we should merge it with
devbranch before fixinguids for good. In fact,forkhelped a lot to understand how we could handle duplication of tasks inside a pipeline, though merkle tree like handling ofuids might be better.This PR uniforms the resulting keys from each orchestrator
.info, e.g.:Now, each orchestrator stores an array of tasks that are inside them, e.g., join(task1, task2) results in [task1, task2] in which each entry is the
taskInvocatorfunction or the orchestrator (join(task1, fork(task2, task3) results in [task1, fork(task2,task3)] where orchestrator is handled as tasks.Also and more importantly fork suffered a major refactoring (to match the other two orchestrators 'shape' and to be able to handle properly
junctionandforkinsidefork. To do so,forkinsideforkneeds to store outer fork downstream tasks, inside outerforkit has to store upstream tasks and downstream tasks for this innerforkand the tasks inside this fork as well. Similar procedures was adopted to handle junction. However, they differ in the waylineageis stored. Whileforkforces duplication of tasks by adding a newuidfor each task inside inner fork and adds a new lineage for each task inside fork (see code here),junctionjust adds downstream and upstream tasks inside thefork, as well as thenewResttasks(tasks after thefork) to thejunctionitself.Also, in this PR a bunch of test cases were added to
./examples/pipelines/testsin order to test these changes in the orchestrators as well as the 'orchestration' between them.I am still working to simplify the code and document it in GSoC17 repo, so don't merge it yet. Just check if you want to change anything or if you can test it (with my
examples/pipelines/testsor other that you find suitable or challenging enough 😄. I also want to stress test it a little more.