Skip to content

Testsuite Ctest CDash and GitLab

Jump to bottom
Mirror Bot edited this page Aug 31, 2026 · 12 revisions

back to home

The testing framework of CFS is composed of the following components

Testsuite

The testsuite is integrated into the main project directly. It must be activated at configuration by turning on BUILD_TESTING. Then you configure and build as usual.

Alternatively, you can use the Testsuite in standalone mode to test an arbitrary installation of openCFS.

Running tests with ctest

To test, go to your build-directory and run

ctest
ctest -E Install # to exclude the install test
ctest -R Mechanic # to include only tests of the mech-PDE
ctest -VV # to get detailed output

You may also descend into the dir of a single test e.g. testsuite/TESTSUIT/Singlefield/Mechanics/cantilevertrans2d and run ctest for running this test.

Creating a new test

The simplest way is to copy an existing test.

  • create a new directory testname for the test and add it to the top cmake-list TESTSUIT/CMakeLists.txt
  • in the new directory you need
    • testname.xml: cfs-input
    • mat.xml: cfs material data file
    • CMakeLists.txt: describing the test
    • testname.h5ref: h5 reference result file (by default all results in the ref file are compared)
  • if you do optimization, run copy_ref_files.py in the directory and commit all the generated .ref files (e.g. ref.info.xml, ref.plot.dat and ref.density.xml)
  • commit your changes to the CFS repository, i.e. in the CFS source directory run git add <your new file>, git commit.

CDash

CDash shows configure/build/test results in a so-called dashboard where they are sorted by day and groups. These are called track or Groups in the CDash admin area (where they can be created, ask admins @ftoth or @Fwein). Our Dashboard is at am-ko.mi.uni-erlangen.de/cdash. The site is publicly available. You are able to submit test results from anywhere, if you have internet access.

To submit to this dashboard there must be a DartConfiguration.tcl in the build or source directory. This file is automatically generated during configure by include(CTest) and properly defining the needed variables before in CMakeLists.txt. Once in place, you can run dashboard tests from the build directory

ctest -D Experimental # run dashboard tests in mode Experimental
# only run some steps and submit to 
ctest -D ExperimentalStart --track Continuous
ctest -D ExperimentalConfigure --track Continuous -V # verbose
ctest -D ExperimentalSubmit --track Continuous
# or in continuous or nightly mode
ctest -D Continuous # tries to update the source if it's up-to-date it stops
ctest -D Nightly # tries to update the source to a defined time

What is submitted is defined by the file Testing/TAG. It contains a number and a track, e.g.

20180420-1309
Experimental

In the submit stage ctest will upload the files Testing/20180420-1309/*.xml to track Experimental. The TAG is set/created in the ctest_start step.

GitLab CI

GitLab CI/CD handles testing and deployment of CFS. All can be viewed in the CI / CD section. Pipelines run for certain events, e.g. for all commits on master or on a branch with an open merge request, and are defined in .gitlab-ci.yml.

If you are working on your branch my-new-feature and are ready for merging to master you should get it tested (e.g. with all our compiler versions). There are currently two types of pipelines:

  • branch pipelines are triggered by the web ui or for pushes to branches called test*, and
  • merge request pipelines run automatically for pushes to branches with open merge requests.

To save resources, branch pipelines do not start all jobs automatically, you will need to trigger them on demand (depending what you want to test). Merge request pipelines should run all necessary jobs automatically. To save computational resources, please set your merge request to Draft when it is not yet ready for testing. In this case, pipelines will be created with all jobs only starting on manual action (clicking the play icon).

The pipeline time is heavily influenced by the build time of the cfsdeps. Therefore, we cache them precompiled on the runners executing the build jobs. We also make use of cache to speed up the CFS build. To get a clean build in the pipeline, use the tag clear-cache on the merge request.

Since the pipeline should pass for all (fast-forward) commits, we have to trigger the pipeline for each commit. This can be done e.g. by resetting your local copy to the previous commit and force pushing to the remote branch via

git reset commit-SHA # specify commit which should be tested
git push --force

Please be careful here since force pushing overwrites your changes on the remote repository! Afterwards, we can step to the next commit by resetting again or we can go back to the initial state (last commit) by

git reset HEAD@{1}
git push --force

Ideally, we push the first commit (git push origin first-commit-SHA:desired-merge-request-branch-name), create the merge-request and then push commits in squence until we have triggered the pipeline for all our commits.

Pipeline configuration

The pipeline defines what should be tested by the runners, which are simply clients connected to the GitLab server (see below). Jobs are assigned to stages which run in succession. One stage may contain multiple jobs, which run in parallel, usually on different runners. What is done in a job is defined in before_script, script and after_script. In CFS we have the build and the test stages. One can assign tags to jobs, to define which runner will pick up which job, i.e. a runner will only pick up a job it it has all the required tags (can be configured online by clicking the edit icon of the runner). To save time the runner uses a caching mechanism, i.e. it zips all files defined in cache. One can use cache:key to define which job uses which cache. However, the cache is usually local to each runner. In order to transfer data between stages one can use artifacts, which are uploaded to the gitlab server. One can define how long they are kept and also download them.

Runners

The system is based on gitlab-runners, which are clients installed on test-machines that wait for jobs sent by the gitlab server, e.g., upon new commits. The runner may use different executors to run jobs. In the simplest case, it simply uses the shell.

We are using the SaaS runners provided by GitLab. For Linux they are based on the docker-executor. The docker images used in our pipeline are collected in the project build-environment. The images are hosted on docker hub.

Self-hosted runners

To add a new runner, one needs to:

  1. Install the runner following the docs
  2. register the runner to the server
  3. start the runner via gitlab-runner start or by starting the service gitlab-runner

Clone this wiki locally