Skip to content

Graduated Testing & Test Numbers

Shelley Lambert edited this page Oct 5, 2022 · 27 revisions

Graduated Testing

Our goal is to employ graduated testing to optimize our use of the machine resources at our disposal. This is accomplished by tagging the tests by level, and choosing to run the 'sanity' level nightly, and the 'sanity' + 'extended' level against the release builds. Each test target may contain many tests, sometimes 1000's.

Some targets are rerun with variations of command-line options, in those cases, the number of tests are recounted, as the new input creates a unique test (a different code path is tested). In the case of system/load tests, some tests are run repeatedly (in some cases 100's to 10,000's of times).

The numbers vary slightly depending on the jdk version and platforms, but for the purpose of this wiki, we will document the average/common numbers and add notes for exceptional differences.

sanity = nightly (Triggered by nightly pipelines)

extended = weekly (Triggered via Jenkins schedule/cron to utilize machine farm during quiet periods on the weekend)

sanity + extended = per release (sanity triggered automatically, extended triggered manually at present, eventually to be triggered by release pipelines when we have sufficient machine resources)

Test Numbers

Test group # of sanity targets # of sanity test cases # of extended targets # of extended test cases # of special targets # of special test cases Platforms Notes
openjdk 8 ~1860 23 ~2260 all jdk_custom and langtools_custom targets allow for running individual test classes
system 89 4454 (bare number) / ~20,000 (with variants) 27 482 (bare number) / ~5000 (with variants) 366 all total: 4936 (bare number)/ 25,040 (with variants) / 31,727,361 (variants with repetitions), extended: 11 targets all jdk impls, 16 are openj9 specific
external 5 ~25,000 12 ~60,000 4 ~40,000 all docker archs x OSs we produce (with exception of windows) sanity targets are microprofile tcks (which are actually being run nightly), extended are functional suites from applications, special are the functional and system test groups containerized
perf 8 25 48 all microbenches are in special.perf
functional 301 ~25,000 364 ~3250 371 all not enabled in automated runs at AdoptOpenJDK, used only in Grinders for now (running nightly at openj9)

This wiki does not include the compliance test numbers in the listing (~70,000-120,000 tests per release, varies by jdk version).

Details on Test Sub-groups

openjdk test group

For the most up-to-date view of these subgroups, please refer to the playlist. The openjdk group is divided into sub-targets, aligned with the jtreg test targets with which many people are already familiar (for example, sanity.openjdk == tier1).

openjdk sanity targets openjdk extended targets Notes
jdk_math, jdk_math_jre, jdk_lang, jdk_io, jdk_net, jdk_nio, jdk_util, jdk_rmi, jdk_security1 jdk_custom, langtools_custom, hotspot_all, hotspot_all_SE100, hotspot_jre, jdk_beans, jdk_other, jdk_security2, jdk_security3, jdk_security4, jdk_text, jdk_time, jdk_management, jdk_jmx, jdk_tools, jdk_jdi, jdk_jfr, jdk_awt (only enabled on linux platforms), jdk_swing (only enabled on linux platforms), jdk_sound (disabled) extended: 2 targets only enabled on linux platforms (awt, swing), 1 target disabled (sound), 3 targets are hotspot specific, not used for openj9 testing (hotspot_all, hotspot_all_SE100, hotspot_jre)

system test group

system sanity targets system extended targets Notes
ClassLoadingTest -- runs 3 tests (3000 times)
MauveSingleThreadLoadTest -- 4619 tests (
500 times)
MauveSingleInvocationLoadTest -- 4619 tests
MauveMultiThreadLoadTest -- 4619 tests (* 1000 times) with at least 3 threads
NioLoadTest - 11 tests (* 10 times), using at least 2 threads
LambdaLoadTest - 14 tests (* 200 times), threads = 2
DaaLoadTest_daa1 - 19 tests (100 times)
DaaLoadTest_daa2 - 28 tests(150 times)
DaaLoadTest_daa3 - 1 test (300 times)
TestJlmLocal - 1
TestJlmRemoteClassAuth - As a server workload : run minimix load run (31 + 4 + 39 + 1) = 75 tests * 300000 times
TestJlmRemoteClassNoAuth - As a server workload : run minimix load run (31 + 4 + 39 + 1) = 75 tests * 300000 times
TestJlmRemoteMemoryAuth - As a server workload : run minimix load run (31 + 4 + 39 + 1) = 75 tests * 300000 times
TestJlmRemoteMemoryNoAuth - As a server workload : run minimix load run (31 + 4 + 39 + 1) = 75 tests * 300000 times
TestJlmRemoteNotifierProxyAuth - As a server workload : run minimix load run (31 + 4 + 39 + 1) = 75 tests * 300000 times
TestJlmRemoteThreadNoAuthTest - As a server workload : run minimix load run (31 + 4 + 39 + 1) = 75 tests * 300000 times
TestJlmRemoteThreadAuth- As a server workload : run minimix load run (31 + 4 + 39 + 1) = 75 tests * 300000 times
TestIBMJlmLocal - 1
TestIBMJlmRemoteClassAuth - As a server workload : run minimix load run (31 + 4 + 39 + 1) = 75 tests * 300000 times
TestIBMJlmRemoteClassNoAuth - As a server workload : run minimix load run (31 + 4 + 39 + 1) = 75 tests * 300000 times
TestIBMJlmRemoteMemoryAuth - As a server workload : run minimix load run (31 + 4 + 39 + 1) = 75 tests * 300000 times
TestIBMJlmRemoteMemoryNoAuth - As a server workload : run minimix load run (31 + 4 + 39 + 1) = 75 tests * 300000 times
TestIBMJlmRemoteNotifierProxyAuth - As a server workload : run minimix load run (31 + 4 + 39 + 1) = 75 tests * 300000 times
TestIBMJlmRemoteThreadNoAuth- As a server workload : run minimix load run (31 + 4 + 39 + 1) = 75 tests * 300000 times
TestIBMJlmRemoteThreadAuth- As a server workload : run minimix load run (31 + 4 + 39 + 1) = 75 tests * 300000 times
Modularity tests - 38

Test running in ConcurrentScavenge mode:
ClassLoadingLoad test - runs 3 test (3000 times)
LambdaLoadTest - 14 (
200 times), threads = 2
mathLoadTest_autosimd - 39 (* 4000 times)
MathLoadTest_bigdecimal = 63 (* 50 times)
DaaLoadTest_daa1 - 19 (*100 times)
DaaLoadTest_daa2 - 28 (150 times)
DaaLoadTest_daa3 - 1 (300 times)
MauveSingleThreadLoadTest -- 4619 (
500 times)
MauveSingleInvocationLoadTest -- 4619
MauveMultiThreadLoadTest -- 4619 (
1000 times) with at least 3 threads
ConcurrentLoadTest -- runs 28 tests (*20 tests)
DirectByteBufferLoadTest -- runs 1 test (DirectMemoryTest)
LangLoadTest -- runs 31 tests (*100 times)
LockingLoadTest -- runs 2 tests
MathLoadTest_all -- 3 tests (*50 times) + 63 tests (50 times) + 39 tests (4000 times)
UtilLoadTest - 4 tests (
500 times), using at elast 2 threads)
DaaLoadTest_all - 48 tests(150 times)
HCRLateAttachWorkload - run minimix load run (31 + 4 + 39 + 1) = 75 tests, with 3 threads
JdiTest - 1 test
HeapHogLoadTest - run minimix load run (31 + 4 + 39 + 1) = 75 tests, 150 times, with at least 2 threads
ObjectTreeLoadTest - 2 tests (
150 times) with 2 threads at least

SharedClassesAPI - run minimix load run (31 + 4 + 39 + 1) = 75 tests, at least 3 threads
SharedClassesWorkload - run minimix load run (31 + 4 + 39 + 1) = 75 tests, * 300 times, with at least 3 threads
SharedClassesWorkloadTest_Softmx_Increase - 1 test
SharedClassesWorkloadTest_Softmx_IncreaseDecrease - 1 test
SharedClassesWorkloadTest_Softmx_Increase_JitAOT - 1 test
SharedClasses.SCM01.SingleCL - 1
SharedClasses.SCM01.MultiCL - 1
SharedClasses.SCM01.MultiThread - 1
SharedClasses.SCM01.MultiThreadMultiCL - 1
SharedClasses.SCM23.MultiCL - 1
SharedClasses.SCM23.SingleCL - 1
SharedClasses.SCM23.MultiThreadMultiCL - 1
SharedClasses.SCM23.MultiThread - 1

Test running in ConcurrentScavenge mode:
MathLoadTest_all -- 3 (*50 times) + 63 (*50 times) + 39 (*4000 times)
DaaLoadTest_all = 48 (*150 times)

external test group

external sanity targets external extended targets Notes
openliberty_microprofile_tck
thorntail_microprofile_tck: 395 testcases (metrics TCK FAT : 151, config TCK FAT : 53, fault-tolerance TCK FAT : 107, restclient TCK FAT : 56, openapi TCK FAT : 28, where FAT is Functional Acceptance Test)
payara_microprofile_tck (derived from Glassfish): 683 testcases (Matrics TCK : 150, Config TCK : 125, Fault tolerance TCK : 186, OpenAPI TCK : 222)
tomee_test
quarkus_quickstarts_test
camel_test
jacoco_test
jenkins_test
kafka_test
wildfly_test
lucene_solr_nightly_smoketest
elasticsearch_test
netty-test
scala_test
tomcat_test
derby_test_junit_all
quarkus_java_test
spring-test
wildfly_test
WycheproofTests

perf test group

perf sanity targets perf extended targets Notes
dacapo-eclipse, dacapo-h2, dacapo-lusearch-fix, idlemicrobenchmark remaining 8 dacapo benchmarks and 18 renaissance benchmarks