Test output gets mixed up when run in parallel

[subhojit777]

Continued from #855

I am using nightwatch-cucumber to run the scenarios. The parallelism is achieved by gulp task runner. I am using gulp.parallel to run the tasks parallely.

gulpfile.js

const gulp = require('gulp');
const gulpNightwatch = require('gulp-nightwatch');

gulp.task('tag1', () => {
  return gulp.src('gulpfile.js')
    .pipe(gulpNightwatch({
      configFile: 'nightwatch.conf.js',
      cliArgs: {
        tag: 'tag1'
      }
    }));
});

gulp.task('tag2', () => {
  return gulp.src('gulpfile.js')
    .pipe(gulpNightwatch({
      configFile: 'nightwatch.conf.js',
      cliArgs: {
        tag: 'tag2'
      }
    }));
});

gulp.task('default', gulp.parallel('tag1', 'tag2'));

[charlierudolph]

For simplicity would the following work? Allow parallelism via a CLI option such as --parallelism <max_concurrent_scenarios>. The default value would be 1 and the parallelism option would also be passed to formatters in order for them to use logic to prevent the output from getting jumbled.

[subhojit777]

Yes that would work. BUT.. if someone is using cucumber-js indirectly (like nightwatch-cucumber) is it going to work. This parallelism you are talking about will only be for cucumber-js, it won't work for others. Nightwatch has its own method of running tests in parallel.

So the conclusion is - we should implement an approach so that it works for cucumber-js as well as any dependent packages.

[subhojit777]

nightwatch-cucumber uses the parallel running strategy of nighwatch, although it didn't work for me

[charlierudolph]

Okay, then we could expose the formatter option that tells cucumber-js that this is part of a parallel run and thus have the formatters act as if cucumber is running in parallel even if the parallelism is provided by something else

[subhojit777]

then we could expose the formatter option that tells cucumber-js that this is part of a parallel run

by expose you mean mentioning in the README?

[charlierudolph]

By expose I mean adding a configuration option that can be used by users and/or tools. It would be documented in one the markdown files linked on the README

[subhojit777]

sounds good. Are you planning to add a CLI option.

[charlierudolph]

There is already a CLI option for format options so it would be added to that

[charlierudolph]

The only formatter this could apply to is the progress formatter (since the pretty formatter was just removed). Instead of printing a character each time a test step finishes, it could be updated to print a single character per test case (analogous to scenario) or print all the step characters when the test case finishes.

[localjo]

I'm also using nightwatch-cucumber and running things in parallel across 14 different environments. The output is not only collated, but it also ends up being way too long and messages about failed tests end up getting lost because they are pushed past the limit of my terminal's history.

A solution for me would be to have the progress indicators completely turned off and to only display the summary for each completed environment, ie;

15 scenarios (1 failed, 14 passed)
113 steps (4 failed, 11 skipped, 98 passed)
1m45.394s

However, I can't find a silent mode or a way to make cucumber stop printing some kind of output to the console.

[fmagaldea]

Hi,

What is the state of this output mix?
As of cucumber-js version 6.0.5 in parallel it's still a mess.

At least, a more detailed doc about formatting restrictions could help.

[aurelien-reeves]

@subhojit777 @localjo nightwatch-cucumber being deprecated in favor of nightwatch-api, and cucumber-js having a good support for parallel executions with recent version, is this still an issue? Could we close it?

@fmagaldea your comment regarding cucumber outpout in parallel still apply? Does it concern nightwatch? If not, could we open a dedicated feature request for this with detailed example of how it is at the moment, and how it may be improved?

[f1ames]

@aurelien-reeves please see #1949. I have reported separate issue as you mentioned since we stumbled upon this (not nightwach related) in our project.

[aurelien-reeves]

Thanks

So, closing this one