Run in a single test, create custom TMT results.yaml with useful logs/artifacts #80

martinpitt · 2022-11-08T15:48:24Z

We are going to call rpminspect_runner.sh just once for a package update. All the extra result parsing and splitting is obsolete, it will be replaced with making rpminspect's new builtin viewer [1] easily accessible from TF results.

[1] https://github.com/rpminspect/rpminspect/tree/main/contrib

I configured https://quay.io/repository/rhn_engineering_mpitt/rpminspect?tab=tags to auto-build on pushes to my fork.

I tested this locally as much as possible with the corresponding pipeline change, replicating a recent real-life run:

git clone -b custom-results-yaml https://github.com/martinpitt/rpminspect-pipeline
cd rpminspect-pipeline
cat <<EOF > env.yaml
PREVIOUS_TAG: f37-updates
TASK_ID: 94237776
RPMINSPECT_PROFILE_NAME: rawhide
REPOSITORY_URL: https://src.fedoraproject.org/rpms/cockpit-podman
CONFIG_BRANCH: rawhide
DEFAULT_RELEASE_STRING: fc37
GIT_COMMIT: f6b1abe13516b37569d2dcd9b718746e97e2e486
EOF

git fetch; git reset --hard origin/custom-results-yaml
rm -rf /var/tmp/tmt/*; tmt run --all --verbose -e @env.yaml

this ran, and as expected failed a few inspections. This has two initial custom logs:

$ cat /var/tmp/tmt/run-001/rpminspect/execute/data/rpminspect/data/results.yaml
- name: /rpminspect
  result: pass
  log:
    - data/rpminspect/output.txt
    - data/rpminspect/data/verbose.log
    - data/rpminspect/data/result.json

Now I want to test this end-to-end via TF, to see how that ends up in results.xml and how we can connect oculus and rpminspect's viewer.html. I hope/think that CI on this PR will do that.

I configured https://quay.io/repository/rhn_engineering_mpitt/rpminspect?tab=tags to auto-build on pushes to my fork.

Wait for Testing Farm to get tmt 1.18.0 to support custom results.yaml (@thrix working on it)
Teach gluetool-modules to consider all log: entries from results.yaml and expose them in results.xml: https://gitlab.com/testing-farm/gluetool-modules/-/merge_requests/282
rebase the above to testing-farm branch
deploy current testing-farm branch (@thrix working on it)
Rebase after some initial cleanups PR Expose result.json and verbose.log as TMT artifacts #83 and fixing rpminspect crash
Reorganize oculus page to show plan and test logs/reproducer etc. on the bottom, with a jump link in the green bar: https://gitlab.com/testing-farm/oculus/-/merge_requests/29
Add rpminspect viewer.html as log artifact
Support HTML artifacts properly in TF viewer: https://gitlab.com/testing-farm/oculus/-/merge_requests/25

Follow-up/ignore:

Rework rpminspect_zuul_runner.sh, this would be completely broken right now.
Clean up $RPMINSPECT_WORKDIR. Not really used any more, but rpminspect_zuul_runner.sh scripts are full of it still

Corresponding pipeline changes are in fedora-ci/rpminspect-pipeline#64 .

martinpitt · 2022-11-09T08:19:33Z

Hmm, there's several blockers here. First of all, tmt's promised "custom results.yaml" does not do anything useful/visible to me. I filed teemtee/tmt#1671 about that. Also, Testing Farm only considers the first entry in log:, and does not plumb through any additional log file. I keep a todo list in the description.

martinpitt · 2022-11-11T05:59:05Z

I sent a testing farm PR to plumb through the additional artifacts to results.xml and thus to the UI: https://gitlab.com/testing-farm/gluetool-modules/-/merge_requests/282 . This needs to land and get rolled out before we can make any progress here.

martinpitt · 2022-11-28T06:51:32Z

@msrb, @thrix : Hmm, it seems the environment files disappeared? My original reference test run as well as a recent one still exist, but the tmt-environment-rpminspect.yaml file mentioned in the reproducer does not exist any more.

It's plausible that the recent rpminspect-runner changes around workdir may have accidentally dropped this file from the artifacts, but why did it also actively disappear from an old result?

I'll try to guess how they should look like, so that I can locally run an rpminspect pipeline. This PR is finally unblocked after last Friday's TF release. 🎉

martinpitt · 2022-11-28T07:06:07Z

Ah, one can see the environment in the tmt log, thus a working file looks like this:

PREVIOUS_TAG: f37-updates
TASK_ID: 93912669
RPMINSPECT_PROFILE_NAME: rawhide
REPOSITORY_URL: https://src.fedoraproject.org/rpms/cockpit
CONFIG_BRANCH: rawhide
DEFAULT_RELEASE_STRING: fc37
GIT_COMMIT: 9fa3d64b44d624a37ab30ff25c56117cb31e86a9

martinpitt · 2022-11-28T08:24:16Z

Unfortunately that old cockpit test cannot be run any more, as rpminspect crashes: rpminspect: libannocheck_init error: version number too small. So I ran a pipeline for this recent successful cockpit-podman bodhi result through jenkins, and it ended up producing this result. This works as intended, I get the extra artifacts and the link to viewer!

We are going to call rpminspect_runner.sh just once for a package update. All the extra result parsing and splitting is obsolete, it will be replaced with making rpminspect's new builtin viewer [1] easily accessible from TF results. [1] https://github.com/rpminspect/rpminspect/tree/main/contrib

tmt 1.18 now supports generating a custom results.yaml [1] . We can use that to declare our verbose.log and results.json as logs/artifacts. These get plumbed through the testing farm API into results.xml, from where they can get picked up by the gating dashboard or the Testing Farm oculus viewer. Move these attachments to the `$TMT_DATA_DIR`, so that they actually get preserved. So far they were in the source git checkout, aka. TMT `tree/` directory, but this is going away [2]. [1] teemtee/tmt#1488 [2] teemtee/tmt#1707

There is just one test now, so this script/step is not necessary any more.

martinpitt · 2022-12-13T10:56:14Z

The rpminspect crash was fixed at last, and depending PR #83 landed. I rebased this, and triggered a new run against voms-api-java-3.3.2-11.fc38, which actually fails rpminspect. Compare the result from current official infra to the result with this PR

msrb · 2022-12-13T11:10:40Z

As discussed on IRC, we don't use the Zuul scripts for anything right now. So I would leave those alone -- no need to update them in this PR.

msrb · 2022-12-13T11:40:55Z

rpminspect_runner.sh

+        ${tests:+--tests=$tests} \
+        ${before_build} \
+        ${after_build_param} \
+        > $verbose_log 2>&1 || rc=$?


Nice trick with the || rc=$? ;)

msrb · 2022-12-13T11:53:35Z

Removing so many lines of code -- amazing! :)

I'd say let's merge this and test the changes end-to-end in the pipeline.

Inspired by fedora-ci/rpminspect-runner#80 this does much the same for installability. Instead of running four separate tests, run all four checks in a single test, then (using the stuff from fedora-ci/mini-tps#18 ) generate the result JSON, put it and the viewer HTML into the test data dir, and write a custom results.yaml pointing to it. This should result in the viewer being easily accessible from the TF results, giving packagers a much nicer view on the results. Signed-off-by: Adam Williamson <awilliam@redhat.com>

This was referenced Nov 8, 2022

Call rpminspect_runner.sh just once fedora-ci/rpminspect-pipeline#64

Merged

how to use custom results.yaml? teemtee/tmt#1671

Open

Custom results implementation teemtee/tmt#1488

Merged

martinpitt mentioned this pull request Nov 29, 2022

Expose result.json and verbose.log as TMT artifacts #83

Merged

martinpitt added 4 commits December 13, 2022 11:37

Drop generate_tmt.sh

e98e11a

There is just one test now, so this script/step is not necessary any more.

Add rpminspect viewer artifact

c7b8860

martinpitt marked this pull request as ready for review December 13, 2022 11:00

msrb reviewed Dec 13, 2022

View reviewed changes

msrb approved these changes Dec 13, 2022

View reviewed changes

msrb merged commit 5e29b5f into fedora-ci:master Dec 13, 2022

martinpitt deleted the custom-results-yaml branch December 13, 2022 12:21

AdamWill mentioned this pull request Dec 6, 2023

Run all checks together and upload custom result viewer fedora-ci/installability-pipeline#25

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Run in a single test, create custom TMT results.yaml with useful logs/artifacts #80

Run in a single test, create custom TMT results.yaml with useful logs/artifacts #80

martinpitt commented Nov 8, 2022 •

edited

Loading

martinpitt commented Nov 9, 2022

martinpitt commented Nov 11, 2022

martinpitt commented Nov 28, 2022

martinpitt commented Nov 28, 2022

martinpitt commented Nov 28, 2022 •

edited

Loading

martinpitt commented Dec 13, 2022

msrb commented Dec 13, 2022

msrb Dec 13, 2022

msrb commented Dec 13, 2022

Run in a single test, create custom TMT results.yaml with useful logs/artifacts #80

Run in a single test, create custom TMT results.yaml with useful logs/artifacts #80

Conversation

martinpitt commented Nov 8, 2022 • edited Loading

martinpitt commented Nov 9, 2022

martinpitt commented Nov 11, 2022

martinpitt commented Nov 28, 2022

martinpitt commented Nov 28, 2022

martinpitt commented Nov 28, 2022 • edited Loading

martinpitt commented Dec 13, 2022

msrb commented Dec 13, 2022

msrb Dec 13, 2022

Choose a reason for hiding this comment

msrb commented Dec 13, 2022

martinpitt commented Nov 8, 2022 •

edited

Loading

martinpitt commented Nov 28, 2022 •

edited

Loading