Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
AFNI Automated testing framework #1
AFNI Automated testing framework
We are implementing automated testing of AFNI programs. The goal is that when any change to AFNI's code is made or proposed on GitHub. Tests are automatically run on CircleCI and the percentage of the code base covered by tests is evaluated and automatically posted to CodeCov and on the github repository itself.
This will provide ongoing assurance that the software is working as expected. Additionally, we are using a public testing framework so that users can determine what tests are being run, propose new tests, and monitor changes in the percentage of the codebase covered by tests. Finally, this framework will facilitate faster development and improved collaboration because it will make it easier to determine if changes to the code have broken previously implemented functionality. We hope that AFNI's adoption of an open continuous testing framework will encourage other major neuroimaging analysis packages to adopt similar practices, improving the quality of neuroimaging analysis software for the community as a whole.
We established the automated building and testing framework on CircleCI and coverage reports posted with CodeCov. Proof of concept tests have been added for 4 AFNI programs, covering 2.5% of AFNI's codebase. Jakub's expertise with docker containers and CircleCI was instrumental in getting testing set up this week and having him on site to work with us saved us several weeks of debugging on our own.
Working on a testing framework for regression testing of scientific pipelines in different environments and adding existing AFNI pipelines.
The goal of the testing framework is to provide tool that helps to create regression tests from the existing pipelines used in publications, etc., and compare the results across multiple versions of various software used in the pipeline. In the future, the framework will allow to run the tests locally.
It will help testing the AFNI pipeline for various versions of the packages and external software (including pipelines that are too long to run for every PR).
referenced this issue
Sep 14, 2018
May be it is just my limited search capabilities while on the phone, but I couldn't determine where are actually the tests and how they run to provide that coverage reporting which is already posted on codecov. Would someone be so kind to point me to the file where tests are run with coverage enabled?