Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Going deeper on fork orchestration #72

Merged
merged 20 commits into from Aug 17, 2017
Merged

Going deeper on fork orchestration #72

merged 20 commits into from Aug 17, 2017

Conversation

@tiagofilipe12
Copy link
Member

@tiagofilipe12 tiagofilipe12 commented Aug 11, 2017

This PR intends to at the very least increase the level of forkception, i.e., the ability to had forks inside forks.
For now it just handles one fork inside another fork, but has no capacity to handle a third fork. Also and importantly, fork can proceed a fork, i.e. we can have something like this:

const pipeline3 = join(
  task0,
  fork(
      fork(
        task1,
        task3
      ),
      task6
  ),
  task5
)
@codecov-io
Copy link

@codecov-io codecov-io commented Aug 11, 2017

Codecov Report

Merging #72 into dev will decrease coverage by 0.04%.
The diff coverage is 87.87%.

Impacted file tree graph

@@            Coverage Diff             @@
##              dev      #72      +/-   ##
==========================================
- Coverage   81.88%   81.83%   -0.05%     
==========================================
  Files          37       37              
  Lines         861      881      +20     
  Branches      102      107       +5     
==========================================
+ Hits          705      721      +16     
- Misses        156      160       +4
Impacted Files Coverage Δ
lib/orchestrators/fork.js 100% <ø> (ø) ⬆️
lib/lifecycle/resolve-input.js 73.33% <ø> (-0.44%) ⬇️
lib/task.js 100% <ø> (ø) ⬆️
lib/orchestrators/join.js 92.3% <87.87%> (-4.19%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7359ced...db98270. Read the comment docs.

@tiagofilipe12 tiagofilipe12 changed the title Going deepet on fork orchestration Going deeper on fork orchestration Aug 11, 2017
// with the joint tasks plus the outermostTask with a new uid
const lineage = [forkee].concat(newUpStreamTasks).concat(newOutermostTasks)
lineageGetter(dispatch, lineage, joinages)
} else { // if a fork is found within another fork

This comment has been minimized.

@thejmazz

thejmazz Aug 12, 2017
Member

is the number of nested forks supported "hardcoded" based on these nested if/else blocks?

@tiagofilipe12
Copy link
Member Author

@tiagofilipe12 tiagofilipe12 commented Aug 17, 2017

This PR now also includes new pipelines for testing multiple inputs as well as new uid generation strategy in which upstream tasks are included in uid creation. Check this GSoC17 entry.

@tiagofilipe12
Copy link
Member Author

@tiagofilipe12 tiagofilipe12 commented Aug 17, 2017

Ok, I think it is ready to merge now, but codecov complains about code that is necessary for some of the tests I ran. Maybe you ( @thejmazz ) can find a way to simplify the code.

@tiagofilipe12
Copy link
Member Author

@tiagofilipe12 tiagofilipe12 commented Aug 17, 2017

Also added a concurrency pipeline example using bluebird Promise.

@thejmazz
Copy link
Member

@thejmazz thejmazz commented Aug 17, 2017

As long as tests are passing I don't mind merging even if code coverage drops. However, if you look at the coverage, it is just that if (forkee.info.props) and else if (forkee.info.type === 'fork') are not being ran in any of the tests. So if you can take a look how to resolve that, it will probably be enough to satisfy codecov.

@thejmazz thejmazz merged commit 8fa4702 into dev Aug 17, 2017
5 checks passed
5 checks passed
codecov/patch 87.87% of diff hit (target 81.88%)
Details
codecov/project Absolute coverage decreased by -0.04% but relative coverage increased by +5.99% compared to 7359ced
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details
security/snyk No new vulnerabilities
Details
@bmpvieira bmpvieira removed the bug label Aug 22, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Linked issues

Successfully merging this pull request may close these issues.

None yet

4 participants