New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Going deeper on fork orchestration #72

Merged
merged 20 commits into from Aug 17, 2017

Conversation

4 participants
@tiagofilipe12
Member

tiagofilipe12 commented Aug 11, 2017

This PR intends to at the very least increase the level of forkception, i.e., the ability to had forks inside forks.
For now it just handles one fork inside another fork, but has no capacity to handle a third fork. Also and importantly, fork can proceed a fork, i.e. we can have something like this:

const pipeline3 = join(
  task0,
  fork(
      fork(
        task1,
        task3
      ),
      task6
  ),
  task5
)

@tiagofilipe12 tiagofilipe12 requested a review from thejmazz Aug 11, 2017

@tiagofilipe12 tiagofilipe12 self-assigned this Aug 11, 2017

@codecov-io

This comment has been minimized.

codecov-io commented Aug 11, 2017

Codecov Report

Merging #72 into dev will decrease coverage by 0.04%.
The diff coverage is 87.87%.

Impacted file tree graph

@@            Coverage Diff             @@
##              dev      #72      +/-   ##
==========================================
- Coverage   81.88%   81.83%   -0.05%     
==========================================
  Files          37       37              
  Lines         861      881      +20     
  Branches      102      107       +5     
==========================================
+ Hits          705      721      +16     
- Misses        156      160       +4
Impacted Files Coverage Δ
lib/orchestrators/fork.js 100% <ø> (ø) ⬆️
lib/lifecycle/resolve-input.js 73.33% <ø> (-0.44%) ⬇️
lib/task.js 100% <ø> (ø) ⬆️
lib/orchestrators/join.js 92.3% <87.87%> (-4.19%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7359ced...db98270. Read the comment docs.

@tiagofilipe12 tiagofilipe12 changed the title from Going deepet on fork orchestration to Going deeper on fork orchestration Aug 11, 2017

// with the joint tasks plus the outermostTask with a new uid
const lineage = [forkee].concat(newUpStreamTasks).concat(newOutermostTasks)
lineageGetter(dispatch, lineage, joinages)
} else { // if a fork is found within another fork

This comment has been minimized.

@thejmazz

thejmazz Aug 12, 2017

Member

is the number of nested forks supported "hardcoded" based on these nested if/else blocks?

@tiagofilipe12

This comment has been minimized.

Member

tiagofilipe12 commented Aug 17, 2017

This PR now also includes new pipelines for testing multiple inputs as well as new uid generation strategy in which upstream tasks are included in uid creation. Check this GSoC17 entry.

@tiagofilipe12

This comment has been minimized.

Member

tiagofilipe12 commented Aug 17, 2017

Ok, I think it is ready to merge now, but codecov complains about code that is necessary for some of the tests I ran. Maybe you ( @thejmazz ) can find a way to simplify the code.

@tiagofilipe12

This comment has been minimized.

Member

tiagofilipe12 commented Aug 17, 2017

Also added a concurrency pipeline example using bluebird Promise.

@thejmazz

This comment has been minimized.

Member

thejmazz commented Aug 17, 2017

As long as tests are passing I don't mind merging even if code coverage drops. However, if you look at the coverage, it is just that if (forkee.info.props) and else if (forkee.info.type === 'fork') are not being ran in any of the tests. So if you can take a look how to resolve that, it will probably be enough to satisfy codecov.

tiagofilipe12 added some commits Aug 17, 2017

@thejmazz thejmazz merged commit 8fa4702 into dev Aug 17, 2017

5 checks passed

codecov/patch 87.87% of diff hit (target 81.88%)
Details
codecov/project Absolute coverage decreased by -0.04% but relative coverage increased by +5.99% compared to 7359ced
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details
security/snyk No new vulnerabilities
Details

@bmpvieira bmpvieira removed the bug label Aug 22, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment