Added jtreg-buffer external suite #5167

judovana · 2024-03-21T15:34:24Z

This PR is adding our buffer of not-yet-if-ever upstreamed jtreg tests, as another external testsuite.
It do pass locally, however in grinder it fails https://ci.adoptium.net/view/Test_grinder/job/Grinder/9229/console Error: cannot find the following tests: jtreg-buffer (note: group target such as sanity is not accepted inside testList) So i guess there is somwehre some another protection.

Except that issue, for which I do not have cure, few questions

external tests do not run with podman-docker docekr wrapper. The traces are quite cryptic, is this already tracked?
- Is there generic/general opinion on moving to root-less container engine (thus podman)?
on grinder jobs, the external tests run as root? I had not found a trace of change of user in logs, so am wondering
- local root-less run proiperly failed for me
is there any way how to copy any testsuite results from container to host?
the tap file is correctly on host, but the suite itself is generating jtrfiles and jtr.xml which I would llike to be able to archive on demand if there occure issues.

judovana · 2024-04-03T07:20:18Z

@sophia-guo maybe you can provide feedback please?

karianna · 2024-04-03T10:08:33Z

@judovana You should run a spell and grammar checker over the comments

smlambert · 2024-04-03T16:52:10Z

PRs to add new tests or features should have an issue that they relate to, as that is what we typically do at the project to socialize new features.

I have created this issue, #5195, where it would say what the new tests do, why they should be added into a certain level, etc (noting we have the habit of adding new tests into the dev level to monitor how they run before 'graduating' tests into other levels).

smlambert · 2024-04-03T17:07:03Z

https://ci.adoptium.net/view/Test_grinder/job/Grinder/9354 to fix the TARGET which needs to match the name in the playlist.xml file (not the name of the directory)

Also will want to address the criteria/questions listed in #5195 ahead of bringing in new tests. We will add this to the contributing new tests documentation going forward, so future contributors have awareness.

external/jtreg-buffer/README.md

external/jtreg-buffer/test.properties

external/jtreg-buffer/test.sh

Co-authored-by: Martijn Verburg <martijnverburg@gmail.com>

if returned, return in 3.9.6

Co-authored-by: Martijn Verburg <martijnverburg@gmail.com>

judovana · 2024-04-04T06:43:05Z

@karianna thanx alot for eyball! All should be done as you requested. Need to try that maven in grinder now.
@smlambert Thanx! I will fix grinder job, and will answer the issue you created.

Can you please answer the remaining questions?

external tests do not run with podman-docker docekr wrapper. The traces are quite cryptic, is this already tracked?
    Is there generic/general opinion on moving to root-less container engine (thus podman)?

In meantime I had found that no, and created #5196 hth

on grinder jobs, the external tests run as root? I had not found a trace of change of user in logs, so am wondering

please?

    local root-less run properly failed for me
is there any way how to copy any testsuite results from container to host?

big please.

the tap file is correctly on host, but the suite itself is generating jtrfiles and jtr.xml which I would like to be able to archive on demand if there occurs issues.

judovana · 2024-04-04T06:51:00Z

https://ci.adoptium.net/view/Test_grinder/job/Grinder/9354 to fix the TARGET which needs to match the name in the playlist.xml file (not the name of the directory)

That do not work for me, not even locally. Locally the
BUILD_LIST=external/jtreg-buffer
TARGET=jtreg-buffer-test
is th eonly viable combination. Other ends with

 Error: cannot find the following tests: ..

Where ther above looks like what you are guggesting.. reruning anyway: https://ci.adoptium.net/view/Test_grinder/job/Grinder/9362/

Also will want to address the criteria/questions listed in #5195 ahead of bringing in new tests. We will add this to the contributing new tests documentation going forward, so future contributors have awareness.

On it. TYVM!

judovana · 2024-04-04T07:12:52Z

https://ci.adoptium.net/view/Test_grinder/job/Grinder/9363/

judovana · 2024-04-04T07:18:14Z

https://ci.adoptium.net/view/Test_grinder/job/Grinder/9363/

ok, so maven version is mandatory. The run now correctly passed. I'm not sure why... I left the target and bulild-list as in original one.. Well Maybe not... Probably harsh morning. Glad it is green.

karianna

Technically OK with me, but Shelly to confirm if this is the right approach

sophia-guo · 2024-04-04T17:36:18Z

on grinder jobs, the external tests run as root? I had not found a trace of change of user in logs

External tests are running inside container as root, which can be updated to create and use unprivileged user if root acess is not required by external tests.

smlambert · 2024-04-04T17:39:06Z

Before we go too far with this PR, I think it would be good to think about these jtreg tests as possible ones to add into the openjdk group, given the intent was to eventually upstream them to that project (based on answers in #5195). I do not see a need to run them as part of the external group, they could go into dev.openjdk

sophia-guo · 2024-04-04T17:54:46Z

is there any way how to copy any testsuite results from container to host?

To make testsuite results available it have to be copied to specific directory. In test.sh you can copy the test result to /testResults , update test.properties with test_results="testResults" and update command in playlist.xml specify --reportsrc . Similar to openliberty-mp-tck

This way all *.xml files are visible to junit plugin. Also check both ARCHIVE_TEST_RESULTS and KEEP_REPORTDIR will keep the results even when the tests pass.

sophia-guo · 2024-04-04T18:14:29Z

I agree with Shelley. dev.openjdk might be a better category to go based on #5195 (comment)

judovana · 2024-04-05T11:59:03Z

is there any way how to copy any testsuite results from container to host?

To make testsuite results available it have to be copied to specific directory. In test.sh you can copy the test result to /testResults , update test.properties with test_results="testResults" and update command in playlist.xml specify --reportsrc . Similar to openliberty-mp-tck

This way all *.xml files are visible to junit plugin. Also check both ARCHIVE_TEST_RESULTS and KEEP_REPORTDIR will keep the results even when the tests pass.

Thanx a lot!

as test_results="testResults" is property, can it have any value, or is it jsut testResults waht is allowed? Also, what is it relative to?

judovana · 2024-04-05T12:19:14Z

Before we go too far with this PR, I think it would be good to think about these jtreg tests as possible ones to add into the openjdk group, given the intent was to eventually upstream them to that project (based on answers in #5195). I do not see a need to run them as part of the external group, they could go into dev.openjdk

Cool! Thanx!

I willexperiment a bit with test_results="testResults" then close this one and do a new into dev.openjdk

judovana · 2024-04-05T13:52:38Z

ok, https://ci.adoptium.net/view/Test_grinder/job/Grinder/9386/ seems to be correctly exporting results. Closing and moving to dev.openjdk

based on: adoptium#5195 adoptium#5167

Added jtreg-buffer external suite

be95549

judovana force-pushed the externalJtregBuffer branch from a5a0e52 to be95549 Compare March 21, 2024 15:35

Run the spell checker on readme

9c40ebb

karianna requested changes Apr 4, 2024

View reviewed changes

judovana and others added 7 commits April 4, 2024 08:25

Update external/jtreg-buffer/README.md

2aecc52

Co-authored-by: Martijn Verburg <martijnverburg@gmail.com>

Update external/jtreg-buffer/README.md

825ef10

Co-authored-by: Martijn Verburg <martijnverburg@gmail.com>

Update external/jtreg-buffer/README.md

8aca196

Co-authored-by: Martijn Verburg <martijnverburg@gmail.com>

Update external/jtreg-buffer/README.md

517f765

Co-authored-by: Martijn Verburg <martijnverburg@gmail.com>

Update external/jtreg-buffer/README.md

8300eb6

Co-authored-by: Martijn Verburg <martijnverburg@gmail.com>

Removed maven at all

ab64983

if returned, return in 3.9.6

Update external/jtreg-buffer/test.sh

ce95bec

Co-authored-by: Martijn Verburg <martijnverburg@gmail.com>

judovana requested a review from karianna April 4, 2024 06:29

Returned maven (in 3.9.6)

2133e23

karianna approved these changes Apr 4, 2024

View reviewed changes

judovana force-pushed the externalJtregBuffer branch from a46b302 to 4f5d9cb Compare April 5, 2024 13:34

Experimental copying of results outside of container

15c6cc9

judovana force-pushed the externalJtregBuffer branch from 4f5d9cb to 15c6cc9 Compare April 5, 2024 13:51

judovana closed this Apr 5, 2024

judovana mentioned this pull request Apr 8, 2024

Consider adding new test material: jtreg-buffer #5195

Open

judovana added a commit to judovana/aqa-tests that referenced this pull request May 10, 2024

Added jtreg-buffer testsuite

6487a36

based on: adoptium#5195 adoptium#5167

judovana mentioned this pull request May 10, 2024

Added jtreg-buffer testsuite #5308

Open

judovana added a commit to judovana/aqa-tests that referenced this pull request May 10, 2024

Added jtreg-buffer testsuite

5e13b5e

based on: adoptium#5195 adoptium#5167

judovana added a commit to judovana/aqa-tests that referenced this pull request May 10, 2024

Added jtreg-buffer testsuite

58208e6

based on: adoptium#5195 adoptium#5167

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added jtreg-buffer external suite #5167

Added jtreg-buffer external suite #5167

judovana commented Mar 21, 2024 •

edited

judovana commented Apr 3, 2024

karianna commented Apr 3, 2024

smlambert commented Apr 3, 2024 •

edited

smlambert commented Apr 3, 2024

judovana commented Apr 4, 2024

judovana commented Apr 4, 2024 •

edited

judovana commented Apr 4, 2024

judovana commented Apr 4, 2024

karianna left a comment

sophia-guo commented Apr 4, 2024

smlambert commented Apr 4, 2024

sophia-guo commented Apr 4, 2024 •

edited

sophia-guo commented Apr 4, 2024

judovana commented Apr 5, 2024

judovana commented Apr 5, 2024

judovana commented Apr 5, 2024 •

edited

Added jtreg-buffer external suite #5167

Added jtreg-buffer external suite #5167

Conversation

judovana commented Mar 21, 2024 • edited

judovana commented Apr 3, 2024

karianna commented Apr 3, 2024

smlambert commented Apr 3, 2024 • edited

smlambert commented Apr 3, 2024

judovana commented Apr 4, 2024

judovana commented Apr 4, 2024 • edited

judovana commented Apr 4, 2024

judovana commented Apr 4, 2024

karianna left a comment

Choose a reason for hiding this comment

sophia-guo commented Apr 4, 2024

smlambert commented Apr 4, 2024

sophia-guo commented Apr 4, 2024 • edited

sophia-guo commented Apr 4, 2024

judovana commented Apr 5, 2024

judovana commented Apr 5, 2024

judovana commented Apr 5, 2024 • edited

judovana commented Mar 21, 2024 •

edited

smlambert commented Apr 3, 2024 •

edited

judovana commented Apr 4, 2024 •

edited

sophia-guo commented Apr 4, 2024 •

edited

judovana commented Apr 5, 2024 •

edited