Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Introduce a Unique ID tracking TestExecutionListener #2619

Closed
7 tasks done
sbrannen opened this issue May 17, 2021 · 2 comments
Closed
7 tasks done

Introduce a Unique ID tracking TestExecutionListener #2619

sbrannen opened this issue May 17, 2021 · 2 comments
Assignees
@sbrannen
Labels
component: Platform theme: reporting type: new feature
Milestone
5.8 RC1

Comments

@sbrannen
Copy link
Member

sbrannen commented May 17, 2021 •
edited
Loading

Overview

Inspired by the simple TestClassTrackingTestExecutionListener that I introduced as a proof of concept in the test suite for the Spring Framework, we should introduce a UniqueIdTrackingListener as an implementation of the TestExecutionListener API for the JUnit Platform.

UniqueIdTrackingListener would track the unique IDs of all "tests" (not containers) that were executed during the execution of test plan. It would track all tests, regardless of the success/failure status of the test, and write the unique IDs to a text file.

The text file can then be used by a build tool or plugin to execute the same set of tests again without having to query the user configuration for the test plan and without having to perform classpath scanning again to discover tests.

This can be useful for test environments such as within a native image -- for example, a GraalVM native image -- in order to rerun the exact same tests from a standard JVM test run within a native image.

Deliverables

  • Introduce UniqueIdTrackingListener in the junit-platform-launcher module.
  • Register the UniqueIdTrackingListener by default via Java's ServiceLoader mechanism.
  • Make the UniqueIdTrackingListener an opt-in feature. If a specific JVM system property has not been set to true, the registered UniqueIdTrackingListener would serve as a no-op listener and not do anything.
  • Make the output directory and output file configurable.
  • Make the output directory convention-based by default, choosing an appropriate build directory for Maven or Gradle, depending on the presence of pom.xml or *.gradle/*.gradle.kts respectively.
  • Document UniqueIdTrackingListener in Javadoc.
  • ❌ Document UniqueIdTrackingListener in the User Guide.
    • Since the User Guide currently has no section dedicated to TestExecutionListener implementations, this will be addressed in a separate issue.
  • Document UniqueIdTrackingListener in the Release Notes.
@sbrannen sbrannen added this to the 5.8 M2 milestone May 17, 2021
@sbrannen sbrannen self-assigned this May 17, 2021
sbrannen added a commit to sbrannen/junit5 that referenced this issue May 25, 2021
@sbrannen
[WIP] Introduce a Unique ID tracking TestExecutionListener
7eee457 
See junit-team#2619
sbrannen added a commit to sbrannen/junit5 that referenced this issue May 25, 2021
@sbrannen
[WIP] Add logging to UniqueIdTrackingListener
924a24f 
See junit-team#2619
sbrannen added a commit to sbrannen/junit5 that referenced this issue May 25, 2021
@sbrannen
[WIP] Polish UniqueIdTrackingListenerTests
594d159 
See junit-team#2619
sbrannen added a commit to sbrannen/junit5 that referenced this issue May 25, 2021
@sbrannen
[WIP] Make the UniqueIdTrackingListener an opt-in feature
f383681 
See junit-team#2619
sbrannen added a commit to sbrannen/junit5 that referenced this issue May 25, 2021
@sbrannen
[WIP] Polish UniqueIdTrackingListener
c960a1b 
See junit-team#2619
sbrannen added a commit to sbrannen/junit5 that referenced this issue May 26, 2021
@sbrannen
[WIP] Polish UniqueIdTrackingListenerTests
3c41e4d 
See junit-team#2619
sbrannen added a commit to sbrannen/junit5 that referenced this issue May 26, 2021
@sbrannen
[WIP] Support custom output filename in UniqueIdTrackingListener
598dce3 
See junit-team#2619
sbrannen added a commit to sbrannen/junit5 that referenced this issue May 27, 2021
@sbrannen
[WIP] Support custom output dir in UniqueIdTrackingListener
64d368b 
See junit-team#2619
sbrannen added a commit to sbrannen/junit5 that referenced this issue May 27, 2021
@sbrannen
[WIP] Support convention-based output dir in UniqueIdTrackingListener
7064132 
See junit-team#2619
@sbrannen sbrannen mentioned this issue May 28, 2021
Closed
7 tasks
@sbrannen sbrannen linked a pull request May 29, 2021 that will close this issue
Introduce a Unique ID tracking TestExecutionListener #2630
Closed
7 tasks
sbrannen added a commit to sbrannen/junit5 that referenced this issue May 31, 2021
@sbrannen
[WIP] Document UniqueIdTrackingListener in Javadoc
9280e06 
See junit-team#2619
sbrannen added a commit to sbrannen/junit5 that referenced this issue May 31, 2021
@sbrannen
[WIP] Document UniqueIdTrackingListener in the Release Notes
5b54944 
See junit-team#2619
sbrannen added a commit that referenced this issue Jun 4, 2021
@sbrannen
Only consider files as potential build scripts in UniqueIdTrackingLis…
3d77425 
…tener

See #2619
@melix melix mentioned this issue Jun 25, 2021
Open
@sbrannen
Copy link
Member Author

Reopening this issue in order to track skipped tests as well.

@sbrannen sbrannen reopened this Jun 25, 2021
@sbrannen
Copy link
Member Author

sbrannen commented Jul 1, 2021

Reopening to support concurrent use cases. See graalvm/native-build-tools#84 for details.

@sbrannen sbrannen reopened this Jul 1, 2021
sbrannen added a commit to sbrannen/junit5 that referenced this issue Jul 1, 2021
@sbrannen
Support concurrent use cases in the UniqueIdTrackingListener
58727cc 
Closes junit-team#2619
@sbrannen sbrannen mentioned this issue Jul 2, 2021
Merged
runningcode pushed a commit to runningcode/junit5 that referenced this issue Feb 15, 2023
@sbrannen @runningcode
Introduce a Unique ID tracking TestExecutionListener
bc52510 
This commit introduces UniqueIdTrackingListener which is a
TestExecutionListener that tracks the unique IDs of all tests that were
executed during the TestPlan and generates a file containing the unique
IDs.

The generated file can be used to rerun those tests -- for example, in
order to run the same set of tests executed on the JVM subsequently
within a GraalVM native image.

See the Javadoc for UniqueIdTrackingListener for configuration options
and details.

Closes junit-team#2619
runningcode pushed a commit to runningcode/junit5 that referenced this issue Feb 15, 2023
@sbrannen @runningcode
Only consider files as potential build scripts in UniqueIdTrackingLis…
3066e1d 
…tener

See junit-team#2619
runningcode pushed a commit to runningcode/junit5 that referenced this issue Feb 15, 2023
@sbrannen @runningcode
Track skipped tests in UniqueIdTrackingListener
930386e 
Closes junit-team#2619
runningcode pushed a commit to runningcode/junit5 that referenced this issue Feb 15, 2023
@sbrannen @runningcode
Support concurrent use cases in the UniqueIdTrackingListener
a9d67f6 
Prior to this commit, the UniqueIdTrackingListener wrote to an output
file with a specific name. This approach works fine when only one
listener is writing the file per test suite. However, when multiple
TestPlans are executed in parallel for the same test suite (for
example, with parallel test execution in a Gradle build), each
UniqueIdTrackingListener will attempt to recreate and write to the
exact same file. The result is that the "last one wins", meaning that
the output file only contains the tracked UIDs for the last
UniqueIdTrackingListener, and all other tracked UIDs are lost.

This commit fixes this by ensuring that each UniqueIdTrackingListener
instance writes to an output file that includes a random number in the
file name, thereby creating one unique output file per listener.

Consequently, consumers of the output files must now read the UIDs from
all such output files. In addition, the output files must be deleted
prior to TestPlan execution since the UniqueIdTrackingListener no
longer overwrites the previous output file.

Closes junit-team#2619
@mpkorstanje mpkorstanje mentioned this issue Mar 24, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sbrannen sbrannen

Labels
component: Platform theme: reporting type: new feature
Projects
None yet
Milestone
5.8 RC1
Development

Successfully merging a pull request may close this issue.

Introduce a Unique ID tracking TestExecutionListener sbrannen/junit5
1 participant
@sbrannen