Enhance TKG repeat feature

[renfeiw]

• add ITERATIONS and AOT_ITERATIONS options
• add option to continue or stop after failure is detected

Signed-off-by: renfeiw renfeiw@ca.ibm.com

[renfeiw]

related: #352

[renfeiw]

test target	options	platform	grinder
floatSanityTests	ITERATIONS=5	x86-64_linux	27751
floatSanityTests	ITERATIONS=5 EXIT_FAILURE=true	x86-64_linux	27752
floatSanityTests	ITERATIONS=5 EXIT_FAILURE=true AOT_ITERATIONS=10	x86-64_linux	27753
floatSanityTests	ITERATIONS=5 EXIT_FAILURE=true AOT_ITERATIONS=10 TEST_FLAG=AOT	x86-64_linux	27754
testSCCMLTests2(aot explicit test)	ITERATIONS=5 EXIT_FAILURE=true AOT_ITERATIONS=10 TEST_FLAG=AOT	x86-64_linux	27755
floatSanityTests	ITERATIONS=10	ppc64le_linux	27778
floatSanityTests	ITERATIONS=10	ppc64_aix	27779
floatSanityTests	ITERATIONS=10	s390x_linux	27780
floatSanityTests	ITERATIONS=10	x86-64_mac	27781
floatSanityTests	ITERATIONS=10	x86-64_windows	27782
floatSanityTests	ITERATIONS=10	s390x_zos	27787
floatSanityTests	ITERATIONS=10	x86-32_windows	27788
sanity		ppc64le_linux	27792
all (BUILD_LIST=TKG/examples/base)	ITERATIONS=6 EXIT_FAILURE=true	ppc64le_linux	27790
all (BUILD_LIST=TKG/examples/base)	ITERATIONS=6 EXIT_FAILURE=false	ppc64le_linux	27791

[llxia]

Could you update TKG_REPEAT and AOT_REPEAT to ITERATIONS and AOT_ITERATIONS in the description? Thanks

[renfeiw]

Could you update TKG_REPEAT and AOT_REPEAT to ITERATIONS and AOT_ITERATIONS in the description? Thanks

Thanks updated.

[llxia]

LGTM

[smlambert]

Thanks @renfeiw, this is nice! I ran an open Grinder to see what the console output looks like.

Note to self: We can likely revert/remove this change in Jenkinsfilebase given it is no longer needed to show iteration progress (and only works for Jenkins environment)

[smlambert]

If I look at the console output, there is a change required in Jenkinsfilebase so that we do not multiply ITERATIONS by ITERATIONS, as I think we both run the make target ITERATIONS times, but also have a for loop in runTest() in Jenkinsfilebase, which we will need to remove or end up running ITERATIONS².

Not sure if we want to merge this first or coordinate a PR to update Jenkinsfilebase and merge together.

[renfeiw]

Yes, it would be better to merge together with this PR: adoptium/aqa-tests#3989

[renfeiw]

grinder: https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5709

[llxia]

hmm, we only have 1 TAP for the Grinder. I cannot tell we had 3 iterations in the TAP file https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5709/tapResults/

[renfeiw]

hmm, we only have 1 TAP for the Grinder. I cannot tell we had 3 iterations in the TAP file https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5709/tapResults/

This is the next step we need to add the iteration info into result summary.

[renfeiw]

Just had a discussion with @llxia We think it's not the right stage to switch over completely to TKG iteration. So I will update the PR to use TKG_ITERATIONS and it will not conflict with Jenkins ITERATIONS.

[renfeiw]

grinder with TKG_ITERATIONS option: https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5712/