Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

EPIC: Provide convenience scripts to verify TAP/metadata file archive is 'good' #3713

Open
1 of 3 tasks
smlambert opened this issue May 31, 2022 · 3 comments
Open
1 of 3 tasks
Assignees

Comments

@smlambert
Copy link
Contributor

smlambert commented May 31, 2022

To alleviate any manual effort, we should provide some convenience/utility scripts that will verify the AQAvit results so that vendors can check that their results are good (and binaries ready to publish).

The features can include:

  • basic checking of TAP files (if any 'not ok' statements present, report on the target name)
  • checking that the release info matches the release file of the binary being published
  • check that the SHAs look 'correct'

This can be done in stages. There can also be enhancements to the Jenkins scripts to gather the Rerun in Grinder with failed targets links for jobs with non-passing status.

@smlambert smlambert self-assigned this Jun 2, 2022
@smlambert
Copy link
Contributor Author

smlambert commented Jun 6, 2022

There are many different pre-built, open-source consumers we can use as the basis for this work.

As a quick and dirty initial start, one can use the prove utility that comes with Perl. In order to assess failures in TAP files in current working directory, one could run:
prove -f -t *.tap
or for verbose mode and searching recursively in a directory called temurin, then this:
prove -v -t *.tap -r ./temurin

If one wanted a pretty output, can install the TAP::Formatter::HTML Perl module and use:
prove -m -Q -t *.tap --formatter=TAP::Formatter::HTML >output.html
for a single html page representing all TAP files assessed (see output.html.txt)

Screen Shot 2022-06-06 at 12 49 08 PM

where the blocks are clickable, so clicking on one of the red blocks links to a section of the page that shows the failure details if they exist in the TAP file, like so:

Screen Shot 2022-06-06 at 4 41 33 PM

@llxia
Copy link
Contributor

llxia commented Sep 7, 2022

Several features that we may want to include:

  • check java version is expected in all TAP files
  • list all level.group per platform to ensure all AQAvit verification required targets are included. This should handle different folder structures and parallelization.
  • list num of passed tests in each level.group to ensure the minimal number of passed tests
  • provide the ability to remove unwanted TAP files. Vendors may run extra test targets and they may not want to include them in the results.zip

@smlambert
Copy link
Contributor Author

Part of this has been delivered via #4277 (thanks Lan!). We can add additional checks into the newly added aqaTap.sh file.

@smlambert smlambert removed their assignment Feb 3, 2023
@smlambert smlambert changed the title Provide convenience scripts to verify TAP/metadata file archive is 'good' EPIC: Provide convenience scripts to verify TAP/metadata file archive is 'good' Mar 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: In Progress
Development

No branches or pull requests

2 participants