Improve test result handling on QGIS CI

[nyalldawson]

QGIS Enhancement: Improve test result handling on QGIS CI

Date 2023/05/02

Author Nyall Dawson (@nyalldawson)

Contact nyall dot dawson at gmail dot com

maintainer @nyalldawson

Version QGIS 3.34

Summary

The QGIS test suite includes thousands of tests which rely on comparing the results of rendering an image with a set of known "good" reference images. The tests are designed to fail if some code changes are made which cause the rendered results to differ by a large amount from the reference image.

Whenever a contributor makes a change to QGIS the test suite is currently run by GitHub workflows. GitHub makes the log of workflows available to these contributors, but the results are very verbose and contain thousands of lines of low level debugging output. Currently, whenever a change is made which causes differences of the rendered image from the reference image and test failures then the contributor must hunt through these logs in order to determine which tests have failed, then manually download a zip artifact from the workflow which contains the actual rendered output image which failed the test. This is an extremely cumbersome process, and is very unfriendly to both experienced and new QGIS contributors. Alternatively, rendered test images are also submitted to a "cdash" instance, but this is not maintained by QGIS contributors, it has proven to be fragile, and also requires advance understanding of the QGIS ci setup in order to be discoverable by contributors.

Proposed Solution

Changes will be made to the GitHub workflows in order to simplify this situation. The goal is that if a rendering test fails, a comment will be automatically added to the Pull Request containing a descriptive message and a link to download the rendered images for debugging. This will ensure that the failure message is immediately visible for all contributors, bypassing the need for them to understand the QGIS workflow setup or wait for someone else to explain the failure to them.

(Ideally, this comment would also directly contain the rendered vs reference images inline, but this may not be possible to achieve within GitHub's available API. See https://stackoverflow.com/questions/68161639/github-actions-post-comment-to-pr-workflow-that-triggered-the-current-workflow, and actions/upload-artifact#50).

It is important to note that there is no guarantee that this project will be successful. Preliminary research did not find any similar working setups in use on other GitHub repositories, and related discussions have only indicated that comment creation containing links to PR artifacts is THEORETICALLY possible. Accordingly this project should be treated as a research project, which may reveal that the proposed solution is NOT possible to implement as described.

Affected Files

• .github/workflows

Backwards Compatibility

If possible, the workflow improvements will be backported to apply to all maintained stable release branches (including LTR).

Votes

(required)

[strk]

I do feel the pain of being lost in the thousands of debug logs, but I was hoping for a solution based on REDUCING those lines to make it clearer what the actual failure was. Doing so would help developers running the testsuite locally. Improving the github-action workflow to make it easier to download the diff image is a great idea, but please keep in mind human testers as well as avoid making running tests more dependent on external infrastructure than already is.

[nyalldawson]

@strk

The two aren't mutually exclusive! Any efforts to make the tests easier to run locally would be very welcome. But this proposal is singularly focused on improving the situation with rendering tests and the CI infrastructure, which is a long standing significant pain point.

[3nids]

I'm quite confident this is feasible :)

Do you plan to also report other failings tests in the comment or only images related test?

What is your approach if there are too many failing tests/images? just showing a few?