Test headless components #5084

andrlos · 2024-02-21T17:09:19Z

err first of all - please ignore the number of commits and if this gets merged, please squash them into one..

this is an attempt of contributing testHeadlessComponents suite into aqa-vit functional tests.

I managed to get a green grinder run for this -> https://ci.adoptium.net/view/Test_grinder/job/Grinder/8902/ .
While I still suspect there is something off with the jtr.xml reports not being correctly propagated to artifacts I believe this can be resolved swiftly in a broader discussion, hence this PR.

This will solve #4875 .

Cheers!

smlambert

Thanks @andrlos - I have added some comments and we will also want to run Grinders for these tests against all platforms/versions to which they apply.

functional/testHeadlessComponents/playlist.xml

functional/testHeadlessComponents/README.md

smlambert · 2024-02-22T01:54:26Z

functional/testHeadlessComponents/README.md

+BOOTJDK_ARCHIVE_DIR - in case the user wants to use arbitrary jdk build, he can provide path to its archive and it will be used in jre execution. Creates $WORKSPACE/bootjdkarchive and downloads latest temurin archive if not set.
+WORKSPACE - directory where the testsuite is going to execute all the tests. ~/workspace by default
+TMPRESULTS - this is a location where the logfiles will be after the testsuite finishes. Same as WORKSPACE by default.
+*JREJDK - This tells the testsuite whether we are testing jre or jdk.


AQAvit has a USE_JRE parameter that could be leveraged, but won't block this PR regarding its use.

oh talking about parameters.. is there anything saying whether the testmachine has a display output? we could use it to enable a few more testscases via XDISPLAY variable.

We set XDISPLAY via the Xvfb Jenkins plugin in our test pipeline code here: https://github.com/adoptium/aqa-tests/blob/master/buildenv/jenkins/JenkinsfileBase#L735-L757 (exceptions for AIX and solaris platforms, where Xvfb plugin does not work, those are set without the plugin as shown in that code snippet).

So, machines at this Jenkins server (ci.adoptium.net) have capability for XDISPLAY.

I should note that for a couple of our Temurin distributions, we only produce a headless build (alpine-linux and riscv). I presume these testcases are still valid against them, but will double-check via Grinders that they run as expected.

and how does the USE_JRE parameter work(what are the possible values)? I need to have JREJDK filled either with a value of "jre" or "jdk" so it knows what kind of behavior it can expect.

USE_JRE is an environment variable, that if set to true, means we TEST_JDK_HOME (the binary that we want to test) to the JRE (rather than the JDK) so that particular tests are run with the JRE rather than the JDK that we pick up

I am happy to bring this PR in as it stands and come back with a follow-up as needed (once we think this through more thoroughly... my worry is whether some vendors do not produce JREs for later versions of Java, you'll still will want to check at the test level whether its available or not).

functional/testHeadlessComponents/build.xml

functional/testHeadlessComponents/test.properties

functional/testHeadlessComponents/playlist.xml

andrlos · 2024-02-22T18:24:24Z

okay, I fixed majority of the mentioned lines, now I just need an answer about the USE_JRE variable adoptium uses.. also added separate commit to limit only on linux, windows and macs.. would have to test and add other platforms to allowed platforms first

andrlos · 2024-02-22T18:35:45Z

huh totally forgot about the mac problem I had since my script uses "declare -A" <- associative array if I remember correctly.. which is not available on mac since the shell it uses is different.. it is either installable by homebrew or I would have to adjust the script heavily.. will have a look at this over the weekend :-)
The second failure is connected to windows which I will address with my buddy Radek on Monday.

smlambert · 2024-02-22T19:02:28Z

For others awareness, a bunch of Grinders were kicked off for this new test against various platforms, see Grinders 8918 through to 8924 as some examples.

See a couple of failures there that warrant investigation, in Grinder/8922 mac machine may have an older version of declare that does not appear to support -A option

13:23:20  + declare -A resArray
13:23:20  /Users/jenkins/workspace/Grinder/aqa-tests/TKG/../functional/testHeadlessComponents/TestHeadlessComponents/testHeadlessComponents.sh: line 187: declare: -A: invalid option
13:23:20  declare: usage: declare [-afFirtx] [-p] [name[=value] ...]

In Grinder/8923, Windows run,

13:23:50  TESTING:
13:23:50  + set -e
13:23:50  + set -o pipefail
13:23:50  + SCRIPT_SOURCE='C:/jenkins/workspace/Grinder/aqa-tests/\functional\testHeadlessComponents\TestHeadlessComponents\testHeadlessComponents.sh'
13:23:50  + '[' -h 'C:/jenkins/workspace/Grinder/aqa-tests/\functional\testHeadlessComponents\TestHeadlessComponents\testHeadlessComponents.sh' ']'
13:23:50  +++ dirname 'C:/jenkins/workspace/Grinder/aqa-tests/\functional\testHeadlessComponents\TestHeadlessComponents\testHeadlessComponents.sh'
13:23:50  ++ cd -P 'C:/jenkins/workspace/Grinder/aqa-tests/\functional\testHeadlessComponents\TestHeadlessComponents'
13:23:50  ++ pwd
13:23:50  + readonly SCRIPT_DIR=/cygdrive/c/jenkins/workspace/Grinder/aqa-tests/functional/testHeadlessComponents/TestHeadlessComponents
13:23:50  + SCRIPT_DIR=/cygdrive/c/jenkins/workspace/Grinder/aqa-tests/functional/testHeadlessComponents/TestHeadlessComponents
13:23:50  ++ uname
13:23:50  + platform=CYGWIN_NT-10.0-20348
13:23:50  + '[' CYGWIN_NT-10.0-20348 == Linux ']'
13:23:50  + '[' CYGWIN_NT-10.0-20348 == Darwin ']'
13:23:50  + '[' CYGWIN_NT-10.0-20348 '!=' CYGWIN_NT-10.0-20348 ']'
13:23:50  + echo 'Unsupported platform'
13:23:50  Unsupported platform
13:23:50  + exit 1
13:23:50  -----------------------------------
13:23:50  TestHeadlessComponents_0_FAILED
13:23:50  -----------------------------------

smlambert · 2024-02-22T19:04:24Z

Great, just saw your comment @andrlos after posting mine. No rush on this, there is great progress. The other platforms looked good, so I am not sure you would need to restrict to the 3 main ones (as you have done in the latest version of the playlist). I will run some tests against other vendors as well, and we can finalize things early next week. Cheers!

andrlos · 2024-02-22T20:13:40Z

@smlambert don't bother trying other platforms than those three for now, they will all get stuck on the following block of code https://github.com/andrlos/TestHeadlessComponents/blob/main/testHeadlessComponents.sh#L21 :-)

smlambert · 2024-02-23T02:21:45Z

re: #5084 (comment) - got it, thanks!

andrlos · 2024-02-26T20:35:04Z

https://ci.adoptium.net/job/Grinder/8952 windows seems to be solved, now lets see about mac..

andrlos · 2024-02-26T21:44:40Z

okay I got it to work on both mac and windows https://ci.adoptium.net/job/Grinder/8961/ <- hopefully a correct grinder run link :D
anyway I will need some time and maybe some more background about how to implement that groovy code for my scenario with DISPLAY to fill in the XDISPLAY variable.. the code itself seems reasonable and exactly what I need, there will just need to be a few more iterations about utilizing it.
To the USE_JRE variable.. we can discuss and brainstorm it more.. the test was originally designed for jre environments and as such it has its own portable that it is downloading (currently one of adoptiums I believe) that it uses for compilation.. but it can utilize any jdk we throw at it by merely filling a variable.

smlambert · 2024-02-27T04:46:33Z

Grinder against an openj9 distribution: https://ci.adoptium.net/job/Grinder/8984

smlambert · 2024-02-27T05:04:09Z

functional/testHeadlessComponents/build.xml

+	<target name="installBrewBash">
+		<exec executable="ruby" failonerror="true">
+			<arg value="-e" />
+			<arg value="'$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)'" />
+		</exec>
+
+		<exec executable="brew" failonerror="true">
+			<arg value="install" />
+			<arg value="bash" />
+		</exec>
+	</target>


Hmm, we don't install prerequisites as part of test execution. Test prereqs are added and managed via ansible playbooks in the infrastructure repo and documented in the Prerequisites doc. For test execution, we want to leave the machines in the same state we found them (cleaning up temp files, not installing new tools), so if we were to consider changing our practices and install something, we would then likely plan to uninstall at the end of a run.

Even though most of our mac nodes are Orka dynamic agents, other vendors may use static / real hardware.

oh sorry.. I did not realize this. Our VMs are always thrashed after the run, so this did not occur to me...

I will try to get rid of the associative array altogether.. it will take me a day or so but will be worth it since it is the only reason to have this dependency

For the record, @sxa thought this prereq was already in our ansible playbooks, but I have not checked, and clearly if you were failing in Grinders, not deployed out to all nodes (or possibly to Orka nodes), but ya, good to try to minimize prereqs where possible

andrlos · 2024-02-28T19:47:48Z

okay.. this is embarassing.. it turns out that the
declare -A
is not such a problem on Mac afterall..
I don't have a clue how come it works now but it just does even on Mac..
All it took was to explicitly run the script with bash..
I tested even on GH actions and even there there is no more a need for a homebrew install (and I am very much sure there was such a need a few months back)
grinder run:
https://ci.adoptium.net/job/Grinder/9013/

smlambert · 2024-02-28T19:54:44Z

declare -A
is not such a problem on Mac afterall..

Sweet!

smlambert · 2024-02-28T20:43:12Z

A few more Grinders for good measure:
https://ci.adoptium.net/job/Grinder/9022

smlambert

LGTM, thanks @andrlos !

jandrlik and others added 25 commits February 6, 2024 22:12

integrating testHeadlessComponents

4907db0

fixed playlist

4a89e2c

fixing the README and execution

f37b577

completing README

03dc3f2

Merge branch 'adoptium:master' into testHeadlessComponents

d79d86b

editting build.xml

8b79d02

fix build.xml

cdafb16

adding dependancy to the dist target

a61807c

fixing path to the runscript

ed1cd29

wrong repo name

b6a211e

trying moving export of OJDK_VERSION_NUMBER over to build.xml

c6ed20c

end property

e571cb7

another attempt of exporting JDK_VERISON_NUMBER

85b302f

NUMER -> NUMBER

67c552f

trying fix reporting

39f65b3

trying fix reporting2

8d21fcf

trying fix reporting3

3772e22

trying fix reporting4

cfff4ec

trying fix reporting5

a34e6cb

trying fix reporting6

c5c9fd7

trying fix reporting7

750e883

trying fix reporting8

b0a3c38

moving THC from external to functional testsuites

bcdd5f5

playlist having wrong path to the TKG resources

6261b41

TEST_STATUS added to the end of the playlist

7f316b0

andrlos mentioned this pull request Feb 21, 2024

Add TestHeadlessComponent suite to AQAvit #4875

Closed

karianna approved these changes Feb 22, 2024

View reviewed changes

smlambert requested changes Feb 22, 2024

View reviewed changes

smlambert reviewed Feb 22, 2024

View reviewed changes

functional/testHeadlessComponents/playlist.xml Outdated Show resolved Hide resolved

revisit after PR comments

88382ef

limiting just to linux/windows/mac archs

09cde8f

jandrlik added 8 commits February 26, 2024 21:39

macFix1

c04a28b

macFix2

c665377

macFix3

be704e6

macFix4

6253b74

macFix5

22260a9

macFix6

1205347

macFix7

3dcb7a4

macFix8

18c6380

karianna requested a review from smlambert February 27, 2024 04:09

smlambert requested changes Feb 27, 2024

View reviewed changes

dependency installation discarding

13e5415

smlambert approved these changes Feb 28, 2024

View reviewed changes

smlambert merged commit 00ea82a into adoptium:master Feb 28, 2024
2 checks passed

smlambert mentioned this pull request Mar 5, 2024

AQAvit Community Call (March 5, 2024 edition) #5090

Closed

smlambert mentioned this pull request May 14, 2024

TestHeadlessComponents_0_FAILED tar: This does not look like a tar archive #5224

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Test headless components #5084

Test headless components #5084

andrlos commented Feb 21, 2024

smlambert left a comment

smlambert Feb 22, 2024

andrlos Feb 22, 2024

smlambert Feb 22, 2024

andrlos Feb 22, 2024

smlambert Feb 22, 2024

andrlos commented Feb 22, 2024

andrlos commented Feb 22, 2024

smlambert commented Feb 22, 2024

smlambert commented Feb 22, 2024

andrlos commented Feb 22, 2024

smlambert commented Feb 23, 2024

andrlos commented Feb 26, 2024

andrlos commented Feb 26, 2024

smlambert commented Feb 27, 2024

smlambert Feb 27, 2024

andrlos Feb 27, 2024

andrlos Feb 27, 2024

smlambert Feb 27, 2024

andrlos commented Feb 28, 2024

smlambert commented Feb 28, 2024

smlambert commented Feb 28, 2024

smlambert left a comment

Test headless components #5084

Test headless components #5084

Conversation

andrlos commented Feb 21, 2024

smlambert left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrlos commented Feb 22, 2024

andrlos commented Feb 22, 2024

smlambert commented Feb 22, 2024

smlambert commented Feb 22, 2024

andrlos commented Feb 22, 2024

smlambert commented Feb 23, 2024

andrlos commented Feb 26, 2024

andrlos commented Feb 26, 2024

smlambert commented Feb 27, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrlos commented Feb 28, 2024

smlambert commented Feb 28, 2024

smlambert commented Feb 28, 2024

smlambert left a comment

Choose a reason for hiding this comment