DynamicTest should be able to provide tags and source information to TestDescriptor

[jhorstmann]

Overview

Currently the source for dynamic tests is a always a MethodSource pointing to the @TestFactory annotated method. In case the dynamic tests are based on external files, like in a cucumber implementation, it should be possible to specify the source.

Example

Given a file "helloword.feature" containing
  """
  Feature: Hello World Feature

    @hello @world
    Scenario: Hello World Scenario
        Given the user name is 'World'
         When the user clicks the hello button
         Then the output is 'Hello World'
  """
  When dynamic tests are created for this feature
  Then the test description should have a file source of "helloworld.feature" line 4
   And the test description should have the tags ["hello", "world"]

Related Issues

• Consider determining line number information for dynamic tests and adding it to ClassTestSource #434
• Enhance DynamicNode with optional TestSource property #1293
• Support classpath resource for custom test source in dynamic tests #1467

Deliverables

• DynamicTest / DynamicContainer should have additional factory methods allowing to specify TestSource
• ❌ DynamicTest / DynamicContainer should have additional factory methods allowing to specify Set<TestTag>
• The above information should be used when creating the DynamicTestTestDescriptor and DynamicContainerTestDescriptor

(The actual creation of the dynamic tests for a cucumber scenario is out of scope)

[sbrannen]

Interesting idea.

However, there are some problems with the proposal.

First and foremost, both TestTag and TestSource reside in the junit-platform-engine artifact and are therefore not visible from the JUnit Jupiter programming model.

With regard to tags, it's less of an issue since tags could be supplied as Set<String> instead of Set<TestTag>; whereas, providing support for something like TestSource would require a new API that can be converted into TestSource.

Another thing to keep in mind is that tags are processed during the overall discovery phase; whereas, dynamic tests and containers are registered during the execution phase. So additional filtering by tags would have to be performed for dynamically registered tests and containers as well.

@junit-team/junit-lambda, thoughts?

[sbrannen]

Related to:

• Consider determining line number information for dynamic tests and adding it to ClassTestSource #434
• Enhance DynamicNode with optional TestSource property #1293

[robeden]

I would also like to be able to have DynamicTests deal with tags. Perhaps the two portions of this request should be broken apart so they could be dealt with separately?

Another thing to keep in mind is that tags are processed during the overall discovery phase; whereas, dynamic tests and containers are registered during the execution phase. So additional filtering by tags would have to be performed for dynamically registered tests and containers as well.

I don't like it as well as the framework dealing with it, but I suppose the TestFactory could do the filtering if it had access to information about what's included/excluded for the run.

[marcphilipp]

Tags would certainly be far less problematic to implement, so splitting would make sense.

@robeden What's your use case for tags on dynamic tests?

[robeden]

We're using JUnit as an engine for our internal system/integration test framework. Most of those tests are being written as shell scripts, which are discovered and handled by a @TestFactory. I based the framework on JUnit, so our shell script-based tests also have tags associated with them. It would be great to allow them to be included/excluded in all the same way normal tests are. It caught me by surprise to see DynamicTest didn't have support for them (as well as display name).

[marcphilipp]

Have you considered writing a TestEngine that supports discovering and executing the shell-script based tests?

[robeden]

No, I haven’t. I found @TestFactory and stopped there. I’ll take a look.

[marcphilipp]

There's a short section in the User Guide about it:
https://junit.org/junit5/docs/current/user-guide/#launcher-api-engines-custom