Allow to run a subset of feature files by passing a comma separated file list

[janpetryk]

Among the many ways of selecting which features should be included (@SelectDirectories, @SelectClasspathResource, @SelectFile etc.) JUnit5 is missing a possibility to pass a comma separated feature file list that would allow to pass the list of the scenarios to be passed as a variable.

This should work similarly to a cucumber.features parameter present in Cucumber API where it accepts a coma separated list of paths (path/to/example.feature, path/to/other.feature), class path resources (classpath:features/example.feature, classpath:features/other.feature) or even a list of specific scenarios with lines where they start (classpath:features/example.feature:5, classpath:features/other.feature:57)

End goal of that would be to be able to pass the list of features as an command line argument:

./gradlew cucumberTest -Djunit.cucumber.files="classpath:features/example.feature, classpath:features/other.feature" that would run only those two features.

Use cases:

• running only a dynamically calculated subset of tests
• re-running failed tests. rerun cucumber plugin produces a file with a list of failed scenarios in a format of

classpath:features/example.feature:5
classpath:features/other.feature:57

Background

There is a @ConfigurationParameter annotation and it is possible to set cucumber.features with JUnit5 and gradle this way but JUnit5 does not play nicely when cucumber.features is combined with @ExcludeTags and @IncludeTags. Skipping directory selectors also results in bugs.

@SelectFiles is not an answer as it doesn't allow passing a list of files as a command line argument.

To properly run a dynamically calculated subset of tests combined with tag selector there should be a new file selector allowing a comma separated list of values.

[sbrannen]

@SelectFiles is not an answer as it doesn't allow passing a list of files as a command line argument.

So, are you saying that @SelectFiles meets your needs when using a @Suite class?

And that you need something analogous to that for use with the ConsoleLauncher?

From the documentation for the ConsoleLauncher:

--select-file=FILE :: Select a file for test discovery. This option can be repeated.

Does that mean you would like to see a --select-files option that accepts a comma-separated list instead of having to specify --select-file multiple times from the command-line?

[janpetryk]

From the documentation for the ConsoleLauncher:

--select-file=FILE :: Select a file for test discovery. This option can be repeated.

I wasn't aware of the console option, I will check it out.

Does that mean you would like to see a --select-files option that accepts a comma-separated list instead of having to specify --select-file multiple times from the command-line?

That would be a start. Going a little bit further I would expect it to accept all of the formats that cucumber can produce as a result of a rerun plugin - comma separated, line break separated and pointing to a specific scenario by a line.

[mpkorstanje]

@janpetryk have you considered using the JUnit Launcher API to rerun test programmatically? This would not require the use of the rerun plugin. See Cucumber JUnit Platform Engine - Rerunning Failed Scenarios

@sbrannen I think there are two needs that could be addressed here

• Supporting line and column number(s) with --select-file and --select-resource to maintain parity with the @SelectFile and @SelectClasspathResource annotations.

• Support for a self contained workflow where failed tests can be rerun with a second invocation of the CLI for example:

<console-launcher>  --select-directory path/to/features --reports-dir=reports
<console-launcher>  --select-ids $(<extract all failed test ids to rerun from the reports dir>)

[github-actions]

If you would like us to be able to process this issue, please provide the requested information. If the information is not provided within the next 3 weeks, we will be unable to proceed and this issue will be closed.

[marcphilipp]

Team decision: Create separate issues for:

1. Add support for passing line and column number to ConsoleLauncher via --select-file and --select-resource using the same syntax that is supported for the URI argument of dynamic tests (?line=...&column=...)
2. Add --select-unique-id support to ConsoleLauncher
3. Explore options for accepting multiple values for all --select options, e.g. as multiple options (whitespace-separated) or comma-separated