jck_custom supports multiple testcases and different testsuite

[sophia-guo]

Enhance jck_custom to support multiple testcases if possible, which would make #4498 more useful feature.

[sophia-guo]

Current jck_custom can't show the testResult ( no testcase information to verify how many and which testcases have run). For example set custom_target as a directory api/java_applet/AppletStub, there is no information of which or how many tests have been run temurin-compliance/job/Grinder/.

[smlambert]

Again, can't help but think there are some options that can be used when running the underlying javatest harness that would make the output more verbose (much like jtreg does).

From Example 5 on page 21 of https://docs.oracle.com/javacomponents/javatest-4-6/cli-user-guide/CLI.pdf for example:

java -jar lib/javatest.jar -config foo.jti -verbose \
-set test.val1 2007 -runtests
This is the same as Example 4 with the exception that it turns on verbose mode and
changes the answer of one of the questions in the configuration. 

What additional info is available in console with -verbose turned on?

[sophia-guo]

This is specific to jck_custom target. I think it is related with migration to JavaTestUtil, which doesn't update the running process of jck_custom. #4512

[sophia-guo]

jck_custom is broken issue opened #5071

[sophia-guo]

Breaking jck_custom to jckruntime_custom, jckcompiler_custom and jckdevtools_custom

[sophia-guo]

Current JavatestUtils.java supports tests with multiple sub-folders ( or multiple testcases) separated by semicolon ( arg tests=sub-folder1;subfolder2 or tests=testcase1;testcase2 or tests=subfolder;testcase ) by changing the tests to be listed with a single space ( the way JCK harness expects). Trying with multiple testcases in grinder only the first testcase can be running successfully. The other tests will be cut as sh command and will not run, which also set the job as successful .

temurin-compliance/job/Grinder/4059/

23:37:07  /bin/sh: 10: api/java_math/BigInteger: not found

The console output shows the custom target has been correctly parsed as expected format tests=api/java_lang/AbstractMethodError/serial/index.html;api/java_math/BigInteger

23:37:06  Running test jckruntime_custom_0 ...
23:37:06  ===============================================
23:37:06  jckruntime_custom_0 Start Time: Thu Feb 22 04:37:06 2024 Epoch Time (ms): 1708576626286
23:37:06  variation: NoOptions
23:37:06  JVM_OPTIONS:  
23:37:06  { \
23:37:06  echo "";	echo "TEST SETUP:"; \
23:37:06  echo "Nothing to be done for setup."; \
23:37:06  mkdir -p "/home/jenkins/workspace/Grinder/aqa-tests/TKG/../TKG/output_17085766269574/jckruntime_custom_0"; \
23:37:06  cd "/home/jenkins/workspace/Grinder/aqa-tests/TKG/../TKG/output_17085766269574/jckruntime_custom_0"; \
23:37:06  echo "";	echo "TESTING:"; \
23:37:06  "/home/jenkins/workspace/Grinder/jdkbinary/j2sdk-image/bin/java" -Djvm.options=" " -Dother.opts="" -cp /home/jenkins/workspace/Grinder/aqa-tests/TKG/../jck/jtrunner/bin JavatestUtil testRoot=/home/jenkins/workspace/Grinder/aqa-tests/TKG/.. jckRoot=/home/jenkins/jck_root/JCK8-unzipped jckversion=jck8d workdir="/home/jenkins/workspace/Grinder/aqa-tests/TKG/../TKG/output_17085766269574/jckruntime_custom_0" configAltPath=/home/jenkins/workspace/Grinder/aqa-tests/TKG/../jck/jtrunner/config testJava="/home/jenkins/workspace/Grinder/jdkbinary/j2sdk-image/bin/java" riJava="/home/jenkins/workspace/Grinder/jdkbinary/j2sdk-image/bin/java" task=cmdfilegen spec=linux_x86-64  testsuite=RUNTIME **tests=api/java_lang/AbstractMethodError/serial/index.html;api/java_math/BigInteger** isCustomTarget="isCustomTarget"; \

[sophia-guo]

As we are running the target by make story text that follows a semicolon (;) will be executed as shell commands or build script. Instead of using ; to separate the testcases we can use a space, which is the exact same way JCK harness expects and is the same way as jdk_custom case.

[sophia-guo]

As JavatestUtil only load testArgs with key,value pairs we can use double quote for tests args.