runtime: Enable customization of parallel workers

[eman2673]

#1044
Tried to cover all the bases but I'm not very familiar with them. However, I'm convinced :)! Awaiting some feedback. I hope something like this can make it into the framework.

[coveralls]

Coverage increased (+0.03%) to 98.164% when pulling d70c4df on eman2673:master into 08d8cac on cucumber:main.

[eman2673]

The Updated Read Me Contents (for convenience)

Parallelization is achieved by having multiple child processes running scenarios.

Customizable work assignment

Cucumber exposes customization of worker assignment via setParallelCanAssign.
The example below overrides the default, () => true which processes test cases
indiscriminately, with a scheme that accepts untagged test cases as well as test cases
where the first tag doesn't match the first tag of any in progress tests.

import { setParallelCanAssign } from '@cucumber/cucumber'
// Accept tests missing tags or no test is running having the same first tag
setParallelCanAssign((pickleInQuestion, picklesInProgress) => _.isEmpty(pickleInQuestion.tags) 
    || _.every(picklesInProgress, ({tags}) => _.isEmpty(tags) || tags[0].name !== picklesInProgress.tags[0].name))

• Example using the handler above
• 2 workers, A and B
• Scenarios tagged as @simple (2 secs) or @complex (3 secs)
• The first tag of the scenarios: [@complex, @complex, @complex, @simple, @simple, @simple]

Time	WIP	Events
0		assigned 1 (@complex) to worker A
0	@complex	skip 2 & 3 (@complex) - assign 4 (@simple) to worker B
2	@complex	skip 2 & 3 (@complex) - assign 5 (@simple) to worker B
3	@simple	assign 2 (@complex) to worker A
4	@complex	skip 3 (@complex) - assign 6 (@simple) to worker B
6		assign 3 (@complex) to worker A
9		done

Note

The coordinator doesn't reorder work as it skips un-assignable tests. Also, it always
returns to the beginning of the unprocessed list when attempting to make assignments
to an idle worker. If there was a worker C in the example above, assignment to worker B
would skip 2 & 3 as shown; then assignment to worker C would also skip 2 & 3 upon
checking the test cases against the handler to determine if they have become assignable.

Custom work assignment prioritizes your definition of assignable work over efficiency.
The exception to this rule is if all remaining work is un-assignable, such that all
workers are idle. In this case Cucumber assigns the next test to the first worker
before continuing to utilize the handler to determine assignable work. Workers become
idle after checking all remaining test cases against the handler. Assignment is
attempted on all idle workers when a busy worker becomes ready.

Coordinator

• load all features, generate test cases
• broadcast test-run-started
• create workers and for each worker
  • send an initialize command
  • when a worker outputs a ready command
    • if there are no more test cases, send a finalize command
    • identify the next processable test case (the next test by default)
    • when there are no processable test cases all idle workers remain idle
    • send a run command with the test case to an idle worker
    • repeat if there are still idle workers
  • if all workers become idle and there are more tests, process the next test case
  • when a worker outputs an event command,
    broadcast the event to the formatters,
    and on test-case-finished update the overall result
• when all workers have exited, broadcast test-run-finished

Worker

• when receiving the initialize command
  • load the support code and runs BeforeAll hooks
  • output the ready command
• when receiving a run command
  • run the given testCase, outputting event commands
  • output the ready command
• when receiving the finalize command
  • run the AfterAll hooks
  • exit

[charlierudolph]

Thank you very much for the contribution @eman2673 and helping get this feature off the ground

[eman2673]

@charlierudolph, thanks for the feedback. I have addressed your concerns. Regarding the warning upon all idle workers, I used console.warn. I'm not sure that's the way to go and if it's not just let me know. I'm thinking the alternative is to broadcast the warning to the formatters (I'll have to figure out how to do that).

[eman2673]

I may have to do some parsing after all. Locally, I get consistent test results but it looks like I can't depend on the server giving me 2 equally responsive workers. This run looks like 1 worker finished all 3 scenarios before the other one was ever ready. Would hate to introduce flaky tests so... WIP

[davidjgoss]

@eman2673 looks like something has gone a bit wrong with the merge from main - there are still some conflicts and the diff is huge (220 files). I'm going to see if I can resolve it locally.

Update: that seems to have done it! I'll get back to you soon with some review feedback.

[eman2673]

@eman2673 looks like something has gone a bit wrong with the merge from main - there are still some conflicts and the diff is huge (220 files). I'm going to see if I can resolve it locally.

Update: that seems to have done it! I'll get back to you soon with some review feedback.

Yeah, I noticed. After resolving all the conflicts, I didn't have it in me to figure out what was going on.

Thanks for taking care of it.

[davidjgoss]

@eman2673 if it's okay I'm going to experiment a bit with the idea of Charlie's above and see where it leads, it should help me understand the code a bit better in any case.

[charlierudolph]

Sorry been so out of touch here. Got some free time and implemented the feature changes I was thinking. As part of it just created a helper that we could export for users to give them a means to get started with this.

Please review @eman2673 @davidjgoss

[mattwynne]

Nice to have you back @charlierudolph! 😁

[davidjgoss]

Looks like there might be some flakiness in the tests here e.g. https://github.com/cucumber/cucumber-js/runs/5244903679?check_suite_focus=true - going to run a few more times and try to see what's going on.

[charlierudolph]

Increased the time to give us a larger buffer which I hope will decrease chance of flakes (at the cost of the test running a bit longer)

[aslakhellesoy]

Hi @eman2673,

Thanks for your making your first contribution to Cucumber, and welcome to the Cucumber committers team! You can now push directly to this repo and all other repos under the cucumber organization! 🍾

In return for this generous offer we hope you will:

• ✅ Continue to use branches and pull requests. When someone on the core team approves a pull request (yours or someone else's), you're welcome to merge it yourself.
• 💚 Commit to setting a good example by following and upholding our code of conduct in your interactions with other collaborators and users.
• 💬 Join the community Slack channel to meet the rest of the team and make yourself at home.
• ℹ️ Don't feel obliged to help, just do what you can if you have the time and the energy.
• 🙋 Ask if you need anything. We're looking for feedback about how to make the project more welcoming, so please tell us!

On behalf of the Cucumber core team,
Aslak Hellesøy
Creator of Cucumber

[eman2673]

@charlierudolph @davidjgoss @mattwynne

Thanks for all the help getting these changes.

[mattwynne]

🎉 you're welcome @eman2673, thanks for all your hard work and patience!

If you're interested in doing any more work on Cucumber just let us know, we can always use more hands on the code. As Aslak said, you can chat with us in our community Slack in the #committers channel 😄

I'm also trying to make myself available for live pairing ⚡ you can book a slot with me here.