Check ProblemLists for incorrect platform entries

[smlambert]

During jdk18 triage, we have seen that there are incorrect entries in the ProblemList file (linux_s390x, linux_ppc64le, etc) so the test will still run, even though we thought it was excluded.

We should also add a readme or a comment at the top of problemlist files to remind people of what formats are accepted by the upstream tools.

Refs:
adoptium/adoptium#133 (comment)
adoptium/adoptium#133 (comment)

[Haroon-Khel]

https://github.com/adoptium/aqa-tests/tree/master/openjdk#exclude-a-testcase for the proper annotation when excluding

[smlambert]

And noting that the problemlist parser (exclude_parser.py) that will run as part of the Auto reenablement pipeline will help this problem.

https://github.com/adoptium/aqa-tests/blob/master/scripts/disabled_tests/exclude_parser.py#L189-L203
https://github.com/adoptium/aqa-tests/tree/master/scripts/disabled_tests#dry-run-to-see-errors-and-warnings

[Nuel-Eneji]

Hello @smlambert trust you're very well.
Can i also take on this issue please.

[Adeosunomoshalewa]

I would like to work on this issue.

[Emidowojo]

Hi @smlambert Can I take this issue?

[smlambert]

I am assigning all 3 of you to this issue.

I think the starting point is to read the documentation at https://github.com/adoptium/aqa-tests/blob/master/scripts/disabled_tests/README.md and try running the disabled_tests scripts and sharing some of your output files (by attaching them as comments to this issue, with any notes you have regarding running the scripts or questions about the documentation).

[Emidowojo]

Hi @smlambert thank you

[Adeosunomoshalewa]

Thank you @smlambert

[Nuel-Eneji]

errors.log

problem_list.json.txt

Greetings @smlambert Hope you're well?
These are my output files from runing the exclude_parser command in the disabled tests
it seems one of the problem files in openjdk/excludes/vendors/azul/ is failing

From the docs, the first command below was reading vendor directory so I used the later command to run my exclude parser test
ls -1dq openjdk/excludes/* | python scripts/disabled_tests/exclude_parser.py > problem_list.json 2> errors.log

find openjdk/excludes/* -name "*.txt" | python3 scripts/disabled_tests/exclude_parser.py > problem_list.json 2> errors.log

Please I'll like to know if I'm on the right track
Thanks

[Nuel-Eneji]

Hello @Emidowojo and @Adeosunomoshalewa
Did you also notice a similar case with the ls -1dq openjdk/excludes/* | python scripts/disabled_tests/exclude_parser.py > problem_list.json 2> errors.log command ?

[Adeosunomoshalewa]

Hi @Nuel-Eneji , haven’t being able to get it from my end , it kept flagging syntax error

[Nuel-Eneji]

Maybe you should also try using this command find openjdk/excludes/* -name "*.txt" | python3 scripts/disabled_tests/exclude_parser.py > problem_list.json 2> errors.log @Adeosunomoshalewa

[Adeosunomoshalewa]

Okay i will do that , Thank you very much

[smlambert]

To your question @Nuel-Eneji re: #3500 (comment)

This is what we hoped to do. Your findings have shown that there is an error in the azul/ProblemList_openjdk8.txt file, and the script has found it:
https://github.com/adoptium/aqa-tests/blob/master/openjdk/excludes/vendors/azul/ProblemList_openjdk8.txt#L74

There should be 3 items per line, the test case to exclude, a link to an issue describing the problem, and the platform on which to apply the exclusion. We see on line 74 that only 1 of those three things are present.

[Nuel-Eneji]

Salutations, @smlambert I've been reading over the code for a while, but I still don't seem to know what to do next, so I would appreciate further advice.
Thanks

[smlambert]

The task here was to run the scripts and identify incorrect entries in our ProblemLists files. You found one, so now we can raise an issue stating the problem and then a PR to fix it (if we know the appropriate way to do so).

Raise an aqa-tests issue entitle "Incorrect formating in vendors/azul/ProblemList_openjdk8.txt file" and describe the issue in the description (you can take the text from my comment above #3500 (comment)). We should then create a PR to fix that line in that file to have 3 sections to it, we need to determine what issue that testcase needs to be excluded under and on what platforms, perhaps defaulting to generic-all.

[Nuel-Eneji]

Thanks @smlambert
I'll get on with it now

[Nuel-Eneji]

Hello, @smlambert I hope everything is good with you.
I did as advised and opened issue #4138.
I'm about to submit a pull request to address the problem.

[Adeosunomoshalewa]

Hi @smlambert , have been working with @Nuel-Eneji on this issue and the output I got from my end.
errors.log

[smlambert]

hi @Adeosunomoshalewa - your errors.log file shows that you do not have Python installed which is needed to run these scripts.

[Adeosunomoshalewa]

Hello @smlambert, after rerunning the text this is what I got
errors.log

[Adeosunomoshalewa]

hello @smlambert, I have rerun the text and have uploaded the file for you to view, kindly acknowledge

[smlambert]

hi @Adeosunomoshalewa - if you look inside the error.log you have produced, you now appear to have python installed, but there is an error.

You get a no such file or directory error. You will have to figure out where the file actually is in order to get a proper result. The error message is telling you that C:\\Users\\USER\\OneDrive\\Documents\\GitHub\\Adeosunomoshalewa\\outreachy\\aqa-tests\\scripts\\disabled_tests\\scripts\\disabled_tests\\exclude_parser.py does not exist. Find where it actually is located in your environment so python3.exe can run it.

python3 : C:\Users\USER\AppData\Local\Microsoft\WindowsApps\python3.exe: can't open file C:\\Users\\USER\\OneDrive\\Documents\\GitHub\\Adeosunomoshalewa\\outreachy\\aqa-tests\\scripts\\disabled_tests\\scripts\\disabled_tests\\exclude_parser.py 

[Errno 2] No such file or directory
...

[Adeosunomoshalewa]

okay @smlambert , will look into it, Thank you

[smlambert]

Marked as complete (by finding and raising #4138).