JUnit 5 @Disabled annotation is ignored

[gaeljw]

👓 What did you see?

We are in the process of moving away from cucumber-junit to cucumber-junit-platform-engine (JUnit 4 to 5).

I just realized that the JUnit 5 @Disabled annotation seem to have no effect on a Cucumber "runner class": the tests are still run.

I reproduced it on the examples project https://github.com/cucumber/cucumber-jvm/tree/main/examples/calculator-java-junit5:

• adding org.junit.jupiter.api.Disabled annotation to RunCucumberTest (needed to add the org.junit.jupiter:junit-jupiter-api dependency in the POM)
• changing a feature file so that there's one error and one scenario failing (easier to see)

✅ What did you expect to see?

I expect the scenarios to not be executed if the "runner class" is flagged as @Disabled

📦 Which tool/library version are you using?

Cucumber 7.26.0

🔬 How could we reproduce it?

I can provide a dedicated separate MRE if needed ;)

📚 Any additional context?

Am I missing something obvious related to JUnit 5 (I'm very new to it) or is this an issue in the cucumber engine?

[mpkorstanje]

Am I missing something obvious related to JUnit 5 (I'm very new to it) or is this an issue in the cucumber engine?

Yes. But don't worry about it. You are in - if not good at least - numerous company. 😉

JUnit 5 = JUnit Platform + JUnit Jupiter + JUnit Vintage. The JUnit Platform is a framework for discovering, selecting and executing tests. The main abstraction is provides are test engines. Jupiter, Vintage and Cucumber are examples of such test engines. The Suite engine responsible for running @Suite annotated class. The suite engine differs from the other engines in that it uses the JUnit Platform to recursively discover and execute tests.

I've put some diagrams in the JUnit Platform Engine docs to clarify

• https://github.com/cucumber/cucumber-jvm/tree/main/cucumber-junit-platform-engine#running-cucumber
• https://github.com/cucumber/cucumber-jvm/tree/main/cucumber-junit-platform-engine#suites-with-different-configurations

From an architecture perspective the @Disabled annotation belongs to the Jupiter Engine rather than the Suite Engine and so it will not have any effect on the Suite engine which is oblivious to the existence of JUnit Jupiter. Which is also why you had to add the dependency in the first place. 😄

Depending on the problem you want to solve you can either:

• Configure the Cucumber Engine to skip features, rules or scenarios tagged with @Disabled. This will skip the test.
• Annotate the @Suite annotated class with @ExcludeTags. Note: This takes a JUnit tag expression, unlike a Cucumber tag expression it does not include the @ symbol. This will remove the test from execution entirely.

There is no annotation to disable a @Suite annotated class entirely, for that we'd have to make a feature request to JUnit. I wouldn't mind doing that, but I'd need to know the exact use-case.

[gaeljw]

Aaaah I knew that I knew nothing about JUnit 5! As always, thanks a lot for the explanation @mpkorstanje 🙏

[gaeljw]

There is no annotation to disable a @suite annotated class entirely, for that we'd have to make a feature request to JUnit. I wouldn't mind doing that, but I'd need to know the exact use-case.

Sorry, I didn't answer this part...

The use case is debatable, hence not sure it's worth asking the feature to JUnit but sharing it anyway:

• we have draft scenarios tagged @draft (cucumber tag) that can be committed in the repo
• once a scenario is ready, we tag it as @valid (cucumber tag)
• we have 2 runner classes: one that targets @valid scenarios and one that targets @draft scenarios ; the latter is kept in the repo for ease of use but should not be run by the CI hence ignoring it from a JUnit POV

We're fine with any of the following solution:

• do not commit the @draft runner class in the first place, it's not a big deal to create it when we need it
• add a "fake" @ignore tag in the @draft runner class so that no scenario is found

[mpkorstanje]

Cheers! Made junit-team/junit-framework#4797 with the suggestion to wait and see if there is interest from other people.

As a solution that is a little bit less harsh than deleting the class, by default JUnit ignores tests classes that don't match this pattern ^(Test.*|.+[.$]Test.*|.*Tests?)$. So calling the draft runner anything else would work. Though it might also be that your build tool overrides that behavior in JUnit.