Below is an example to run all sanity.functional tests against a JDK8 Linux x86-64 cmprssptrs OpenJ9 SDK:
cd openj9/test/TestConfig export TEST_JDK_HOME=/my/openj9/jdk make -f run_configure.mk // generates makefiles make compile // downloads test related material/libs // and compiles test material make _sanity.functional // runs tests
Please read OpenJ9 Test User Guide for details and other examples. For example, if you wish to test a JDK11 version, you would export JDK_VERSION=11 (default JDK_VERSION=8).
Please read Prerequisites.md for details on what tools should be installed on your test machine to run tests.
While the OpenJ9 Test User Guide gives a more complete list of OpenJ9 test use cases, there are some frequently asked questions or common use cases by OpenJ9 developers listed below.
1) How to compile tests in selected directories?
make compile compiles all tests. This is the safest way
to ensure all the test code needed has been compiled. However, there is a
way to shortcut the compilation process to reduce compilation time. If
BUILD_LIST is set,
make compile will only compile the folder names
names that match within
export BUILD_LIST=functional/TestUtilities,functional/Java8andUp make compile
2) How to add a test?
Add FV (functional) test
Adding a testNG test as an example:
- adding a single test class to an existing directory
- update testng.xml to add the test class to a existing or create a new
- If the new is created in testng.xml, playlist.xml should
be updated to add the new based on playlist.xsd
Supported test groups are
functional|system|openjdk|external|perf|jck. It is required to provide one group per test in playlist.xml.
- adding additional new test methods for new Java10 functionality
- test should be automatically picked up
Add external test
Please refer to the video and tutorial that describes how to add container-based 3rd party application tests (run inside of Docker images). These tests are added and run in the automated test builds at the AdoptOpenJDK project.
3) How to disable a test?
In playlist.xml, to disable a test target, add
<disabled>Reason for disabling test, should include issue number</disabled>
<test> element that you want to disable.
- Disable an individual test class
- testNG test
add a line to
TestConfig/resources/excludes/latest_exclude_$(JDK_VERSION).txtfile with issue number and specific specs to disable
- testNG test add a line to
org.openj9.test.java.lang.management.TestOperatingSystemMXBean 123 linux_x86-64
4) How to execute a different group of tests?
Test can be run with different levels, groups and types or combination of two (i.e., level.group, level.type, group.type) or three (i.e., level.group.type)
Supported levels are
Supported groups are
Supported groups are
make _sanity make _functional make _extended.perf make _sanity.native make _extended.functional.native
5) How to execute disabled tests?
If a test is disabled using
<disabled> tag in playlist.xml, it can be executed through specifying the test target or adding
disabled in front of regular target.
make _testA // testA has <disabled> tag in playlist.xml make _disabled.sanity.functional make _disabled.extended
Disabled tests and reasons can also be printed through adding
echo.disabled in front of regular target.
make _echo.disabled.testA make _echo.disabled.sanity.functional make _echo.disabled.extended
6) How to execute a directory of tests?
The example below executes all of the sanity tests found within the JIT_Test directory
make -C ../functional/JIT_Test -f autoGen.mk _sanity
cd ../functional/JIT_Test; make -f autoGen.mk _sanity
7) How to run an individual JCK?
Please read How-to Run customized JCK test targets for details.
8) How to run the test with different
User can run tests against different jdk version and/or jdk implementation. While the default values of these variables match a typical use case for OpenJ9 developers, there are also many cases where developers need to verify features for a specific version or compare behaviour against a particular implementation.
There is no extra step needed. By default, AUTO_DETECT is turned on, and the test framework will auto detect SPEC, JDK_IMPL, and JDK_VERSION. Please read Configure environment for details and examples.
9) How to interpret test results?
- test results summary
At the end of each run, test results summary will be printed:
TEST TARGETS SUMMARY +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ DISABLED test targets: cmdLineTester_pltest_tty_extended_0 cmdLineTester_pltest_numcpus_notBound_0 ... PASSED test targets: cmdLineTester_javaAssertions_0 cmdLineTester_LazyClassLoading_0 ... FAILED test targets: TestAttachAPIEnabling_SE80_0 TestFileLocking_SE80_0 TOTAL: 91 EXECUTED: 84 PASSED: 82 FAILED: 2 DISABLED: 7 SKIPPED: 0 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
You can find the failed test output in console output.
- TAP result
A simple standardized TAP output is produced at the end of a test run, to provide developers with a convenient summary of the test results. It is also necessary as the tests used to verify OpenJ9 use a variety of test output formats. This summary is a way to standardize the output which allows CI tools to present results in a common way.
If a test is skipped, it means that this test cannot be run on this platform due to jvm options, platform requirements and/or test capabilities.
10) How to rerun failed tests?
failed.mk will be generated if there is any failed test target.
We can rerun failed tests as following:
failed.mk will be over-written each test run. If you want to
'save it', you can make a copy of the generated