The OS-autoinst project aims at providing a means to run fully automated tests. Especially to run tests of basic and low-level operating system components such as bootloader, kernel, installer and upgrade, which can not easily and safely be tested with other automated testing frameworks. However, it can just as well be used to test firefox and openoffice operation on top of a newly installed OS.
os-autoinst can be executed alone, but is currently designed to be executed together with openQA, the web user interface that allows to run more than one os-autoinst instance at the same time.
More information on os-autoinst and openQA can be found on http://os-autoinst.github.io/openQA/
To install os-autoinst, you can see INSTALL
To execute an instance of os-autoinst, we need to create file named vars.json, that we use to store the values of the different variables that will configure and drive the behavior of the application.
There are some variables used by os-autoinst itself and other that are used by the tests. A minimal vars.json file can be:
{
"DISTRI" : "opensuse",
"CASEDIR" : "/full/path/for/tests",
"NAME" : "test-name",
"ISO" : "/full/path/for/iso",
"VNC" : "91",
"BACKEND" : "qemu",
"DESKTOP" : "kde"
}Be advised that the file vars.json is read and also written by isotovideo so make sure to backup handcrafted versions of this file.
Now we can launch the application with:
$ ../os-autoinst/isotovideo
and we can use VNC to connect to the QEMU instance:
$ vncviewer localhost:91 -ViewOnly -Shared
Run isotovideo with the environment variable RUN_VNCVIEWER set to autostart
a VNC viewer on the right port.
Run isotovideo with the environment variable RUN_DEBUGVIEWER to start the
internal debug screenshot viewer updated with an always recent screenshot of
the test run.
If you want to contribute to this project, please clone and send git-pull requests via https://github.com/os-autoinst/os-autoinst.
More information on the contribution can be found on http://os-autoinst.github.io/openQA/contact/, too.
Issues are tracked on https://progress.opensuse.org/projects/openqav3/
For an overview of the architecture, see doc/architecture.md.
-
Every commit is checked by Travis CI as soon as you create a pull request but you should run the os-autoinst tests locally. Checkout the build instructions for further details.
-
For git commit messages use the rules stated on How to Write a Git Commit Message as a reference
-
Every pull request is reviewed in a peer review to give feedback on possible implications and how we can help each other to improve
If this is too much hassle for you feel free to provide incomplete pull requests for consideration or create an issue with a code change proposal.
The required dependencies are delcared in dependencies.yaml. (The names listed within
that file are specific to openSUSE but can be easily transferred to other distributions.)
Create a build directory outside of the source directory. The following commands need to be invoked within that directory.
Configure build:
cmake $path_to_os_autoinst_checkout
You can specify any of the standard CMake variables, e.g. -DCMAKE_BUILD_TYPE=Debug
and -DCMAKE_INSTALL_PREFIX=/custom/install/prefix.
The following examples assume that GNU Make is used. It is possible to generate for
a different build tool by adding e.g. -G Ninja to the CMake arguments.
Build executables and libraries:
make symlinks
This target also creates symlinks of the built executables and libraries within the
source directory so isotovideo can find them.
Run all tests:
make test
Run all Perl tests (*.t files found within the t directory):
make test-perl-testsuite
Run individual tests by specifying them explicitly:
make test-perl-testsuite TESTS="15-logging.t 28-signalblocker.t"
By default CTest is invoked in verbose mode because prove already provides condensed
output. Add -DVERBOSE_CTEST=OFF to the CMake arguments to avoid that.
Add additional arguments to the prove invocation, e.g. enable verbose output:
make test-perl-testsuite PROVE_ARGS=-v
Gather coverage data while running tests:
make test-perl-testsuite WITH_COVER_OPTIONS=1
Generate a coverage report from the gathered coverage data:
make coverage
If no coverage data has been gathered so far the coverage target will invoke the
testsuite automatically.
Reset gathered coverage data:
make coverage-reset
Install files for packaging:
make install DESTDIR=…
Further notes:
-
It is also possible to run
ctestwithin the build directory directly instead of using the mentioned targets. -
All mentioned variables to influence the test execution (
TESTS,WITH_COVER_OPTIONS, …) can be combined and can also be used with thecoveragetarget.