Skip to content

TriBITS CDash Dashboard

Roscoe A. Bartlett edited this page Aug 13, 2021 · 21 revisions

Overview

TriBITs is currently being tested using at using the GitHub Actions with test results submitted to the CDash site:

TriBITS is tested for a few different versions of CMake, compilers, Python, and other variations post to the following CDash groups:

  • Nightly: Nightly builds posted every day regardless if there have been any repository activity that day [1].
  • Continuous: Pushes to 'master' [2].
  • Pull Request: Pull requests against 'master' with pr-<id>_ prepended to the build names for the PR ID.
  • Experimental: Any other submits of TriBITS test results using, for example, make dashboard on local machines.

For more details on the testing that is done, see the active GitHub Actions drivers.

  • [1]: Nightly builds primarily test that underlying infrastructure to ensure that everything is working so that when PRs and pushes are done, it will work when needed.
  • [2]: Even through each branch on a PR is tested before merging to 'master', it is still important to test when PR branches are merged to 'master' because two independent PRs that are tested at the same time can pass all of the tests independently but fail when merged together (i.e. a violation of the "additive test assumption of branches").

GitHub Actions Pull Request Testing

Automated testing of TriBITS Pull Requests is currently performed using GitHub Actions as described above. TriBITS has implemented bidirectional links from the GitHub actions jobs to the results on CDash and from the builds on CDash back to the GitHub Actions. The builds on CDash for a single PR testing iteration look like:

Single PR Iteration Builds on CDash

(NOTE: The current set of builds being done may be different that what is shown above.)

Revision column on CDash build

As shown above, each build on CDash includes a "Revision" field that gives the shortened Git SHA1 for the commit being tested. In the case of a PR, this is the SHA1 for a merge commit merging the PR topic branch into the target branch (i.e. 'master'). If one clicks on that SHA1 on CDash it takes you to a CDash page like this that gives the full SHA1 and is hyperlinked to the commits on GitHub. Amazingly, GitHub stores these merge commits so they can be examined on GitHub.

Links from CDash builds back to GitHub PR

As shown above, each build on CDash shows the icon PR Link Icon that provides a link back to the PR (PR #401 in this case).

Links from GitHub Actions Jobs to results on CDash

In addition, the GitHub Actions Jobs for a PR testing iteration provide hyperlinks to the build result on CDash under the "URL to results on CDash" step of the job as shown below.


Screenshots of the GitHub Actions Job hyperlink to results on CDash

On each passing GitHub Actions testing job, there is a "URL to results on CDash" step that contains a link to the results on CDash as shown below:

Screenshot of the GitHub Actions Job "URL to results on CDash" hyperlink

This hyperlink takes you to a CDash build view as shown below:

CDash results for build pr-401_tribits_cmake-3.17.5_makefiles_python-2.7_g++-8

However, when there is a failure, the "URL to results on CDash" step is not run but a hyperlink to the results on CDash is also provided at the end of the "Run configure, build, test and submit to CDash" step as shown below:

Screenshot of the GitHub Actions job "Run configure, build, test and submit to CDash" hyperlink for a failure

Query of all builds for a given PR on CDash

As shown above, the builds for any given PR <pr-id> (401 in this case) is given the build-name prefix pr-<pr-id>_. This makes it easy to create a CDash query locating all of the configure, build, and test results associated with a PR using a query URL like this for PR #401 (just replace 401 in the Build Name field with the <pr-id> for the PR of interest). This shows results like shown below.

Screenshot of all build and test results on CDash for PR #410

Screenshot of all build and test results on CDash for PR #410

Query of all GitHub Actions Test Jobs for a given PR

One can view all of the GitHub Actions PR testing iterations for a given PR using a GitHub Actions Query like this for PR #401 with topic branch 154-improve-cdash-github-interaction with results as shown below.

Screenshot of query for all GitHub Actions test iterations for a given PR branch

It is not too hard to match up those GitHub Actions PR testing iterations and the builds on CDash to go back in history. But again, going from a given GitHub Action job to results on CDash is a direct hyperlink. It is just that going from an arbitrary CDash build to its driving GitHub Actions Job is not as direct. (Adding direct links for that would be a feature request.)

Implementation details

For details on the implementation of this, see GitHub Actions drivers, PR #389, #401, and other commits related to Issues #363 and #154.