Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test discovery: Make classpath scanning fallback configurable #4017

Open
wants to merge 5 commits into
base: master
Choose a base branch
from

Conversation

OliverO2
Copy link
Contributor

Fixes #3973.

Introduces a system property
"kotest.framework.discovery.classpath.scanning.enabled",
defaulting to "true".

This exists to work around a Gradle bug which occurs with JunitPlatform
and maxParallelForks > 1 (process parallelization). In this case,
depending on the number of CPU cores, Gradle may invoke all but one
Kotest launchers with a non-empty list of class selectors, and the
remaining Kotest launcher with an empty list of class selectors,
triggering a classpath scan in Kotest. In that case, tests will execute
twice (discovered once via a class selector, a second time via the
classpath scan).

Also makes Kotest-internal debug logging multi-process-aware,
naming log files like "kotest-PID.log", containing the respective
process id.

@OliverO2 OliverO2 requested a review from sksamuel May 13, 2024 19:26
@sksamuel
Copy link
Member

Fixes #3973.

Introduces a system property "kotest.framework.discovery.classpath.scanning.enabled", defaulting to "true".

This exists to work around a Gradle bug which occurs with JunitPlatform and maxParallelForks > 1 (process parallelization). In this case, depending on the number of CPU cores, Gradle may invoke all but one Kotest launchers with a non-empty list of class selectors, and the remaining Kotest launcher with an empty list of class selectors, triggering a classpath scan in Kotest. In that case, tests will execute twice (discovered once via a class selector, a second time via the classpath scan).

Also makes Kotest-internal debug logging multi-process-aware, naming log files like "kotest-PID.log", containing the respective process id.

If it sends all but one a non empty list, how does the maxParallelForks do anything? Does it just spawn multiple kotest's that don't do anything?

@OliverO2
Copy link
Contributor Author

If it sends all but one a non empty list, how does the maxParallelForks do anything? Does it just spawn multiple kotest's that don't do anything?

It spawns multiple Kotests (each in its own JVM). Then almost all of them get a non-empty list of classes, so these Kotest's do their stuff. Just (the last?) one gets an empty list and (with this PR) does nothing (instead of doing a classpath scan and running all tests again). Seems like a classical "off by one" error in Gradle's Java test task.

More detailed description here: #3973 (comment)

Considerations here: #3973 (comment)

#3973 shows how the drama unfolds.

@Kantis
Copy link
Member

Kantis commented May 20, 2024

Should we consider moving to making classpath scanning disabled unless the ClasspathRootSelector is included? Perhaps adding a fat warning when there's no selectors at all stating that this default will be changing in 6.0?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Tests get run multiple times when using maxParallelForks
3 participants