openQA installation guide
- Basic configuration
- Run the web UI
- Run workers
- Where to now?
- Advanced configuration
- Auditing - tracking openQA changes
- Filesystem Layout
openQA is an automated test tool that makes it possible to test the whole installation process of an operating system. It’s free software released under the GPLv2 license. The source code and documentation are hosted in the os-autoinst organization on GitHub.
This document provides the information needed to install and setup the tool, as well as information useful for everyday administration of the system. It’s assumed that the reader is already familiar with openQA and has already read the Starter Guide, available at the official repository.
Keep in mind that there can be disruptive changes between openQA versions. You need to be sure that the webui and the worker that you are using have the same version number or, at least, are compatible.
For example, the package distributed with openSUSE Leap 42.3 is not compatible with the version on Tumbleweed. And the package distributed with Tumbleweed may not be compatible with the version in the development package.
Installation from distribution packages
The easiest way to install openQA is from distribution packages.
For openSUSE, packages are available for Leap 42.3 and later.
For Fedora, packages are available in the official repositories for Fedora 23 and later.
You can install the packages using these commands.
# openSUSE Leap 42.3+ zypper in openQA # Fedora 23+ dnf install openqa openqa-httpd
Installation from development versions of packages
You can find the development version of openQA in OBS in the openQA:devel repository.
To add the development repository to your system, you can use these commands.
# openSUSE Tumbleweed zypper ar -f obs://devel:openQA/openSUSE_Tumbleweed devel-openQA # openSUSE Leap 15 zypper addrepo https://download.opensuse.org/repositories/devel:/openQA/openSUSE_Leap_15.0/ # openSUSE Leap 42.3 zypper ar -f obs://devel:openQA/openSUSE_Leap_42.3 devel-openQA zypper ar -f obs://devel:openQA:Leap:42.3/openSUSE_Leap_42.3 devel-openQA-perl-modules
Then you can install them using this command.
# all openSUSE zypper install devel-openQA:openQA
It is required to run openQA behind an http proxy (apache, nginx, etc..). See the openqa.conf.template config file in /etc/apache2/vhosts.d (openSUSE) or /etc/httpd/conf.d (Fedora). To make everything work correctly on openSUSE, you need to enable the 'headers', 'proxy', 'proxy_http' and 'proxy_wstunnel' modules using the command 'a2enmod'. This is not necessary on Fedora.
# openSUSE Only # You can check what modules are enabled by using 'a2enmod -l' a2enmod headers a2enmod proxy a2enmod proxy_http a2enmod proxy_wstunnel
For a basic setup, you can copy openqa.conf.template to openqa.conf and modify the ServerName if required setting. This will direct all HTTP traffic to openQA.
cp /etc/apache2/vhosts.d/openqa.conf.template /etc/apache2/vhosts.d/openqa.conf
By default openQA expects to be run with HTTPS. The openqa-ssl.conf.template Apache config file is available as a base for creating the Apache config; you can copy it to openqa-ssl.conf and uncomment any lines you like, then ensure a key and certificate are installed to the appropriate location (depending on distribution and whether you uncommented the lines for key and cert location in the config file). On openSUSE, you should also add SSL to the APACHE_SERVER_FLAGS so it looks like this in /etc/sysconfig/apache2:
If you don’t have a TLS/SSL certificate for your host you must turn HTTPS off. You can do that in /etc/openqa/openqa.ini:
[openid] httpsonly = 0
Since version 4.5.1512500474.437cc1c7 of openQA, PostgreSQL is used as the database.
To configure access to the database in openQA, edit /etc/openqa/database.ini and change the settings in the [production] section.
The dsn value format technically depends on the database type and is documented for PostgreSQL at DBD::Pg
Example for connecting to local PostgreSQL database
[production] dsn = dbi:Pg:dbname=openqa
Example for connecting to remote PostgreSQL database
[production] dsn = dbi:Pg:dbname=openqa;host=db.example.org user = openqa password = somepassword
For older versions of openQA, you can migrate from SQLite to PostgreSQL according to DB migration from SQLite to PostgreSQL
OpenQA supports three different authentication methods - OpenID (default), iChain and Fake. See auth section in /etc/openqa/openqa.ini.
[auth] # method name is case sensitive! method = OpenID|iChain|Fake
Independently of method used, the first user that logs in (if there is no admin yet) will automatically get administrator rights!
By default openQA uses OpenID with opensuse.org as OpenID provider. OpenID method has its own openid section in /etc/openqa/openqa.ini:
[openid] ## base url for openid provider provider = https://www.opensuse.org/openid/user/ ## enforce redirect back to https httpsonly = 1
OpenQA supports only OpenID version up to 2.0. Newer OpenID-Connect and OAuth is not supported currently.
Use only if you use iChain (NetIQ Access Manager) proxy on your hosting server.
For development purposes only! Fake authentication bypass any authentication and automatically allow any login requests as 'Demo user' with administrator privileges and without password. To ease worker testing, API key and secret is created (or updated) with validity of one day during login. You can then use following as /etc/openqa/client.conf:
[localhost] key = 1234567890ABCDEF secret = 1234567890ABCDEF
If you switch authentication method from Fake to any other, review your API keys! You may be vulnerable for up to a day until Fake API key expires.
Run the web UI
systemctl start postgresql systemctl start openqa-gru systemctl start openqa-webui # openSUSE systemctl restart apache2 # Fedora # for now this is necessary to allow Apache to connect to openQA setsebool -P httpd_can_network_connect 1 systemctl restart httpd
The openQA web UI should be available on http://localhost/ now. To ensure openQA runs on each boot, you should also systemctl enable the same services.
systemctl enable postgresql systemctl enable openqa-gru systemctl enable openqa-webui
Workers are processes running virtual machines to perform the actual testing. They are distributed as a separate package and can be installed on multiple machines but still using only one WebUI.
# openSUSE zypper in openQA-worker # Fedora dnf install openqa-worker
To allow workers to access your instance, you need to log into openQA as operator and create a pair of API key and secret. Once you are logged in, in the top right corner, is the user menu, follow the link 'manage API keys'. Click the 'create' button to generate key and secret. There is also a script available for creating an admin user and an API key+secret pair non-interactively, /usr/share/openqa/script/create_admin, which can be useful for scripted deployments of openQA. Copy and paste the key and secret into /etc/openqa/client.conf on the machine(s) where the worker is installed. Make sure to put in a section reflecting your webserver URL. In the simplest case, your client.conf may look like this:
[localhost] key = 1234567890ABCDEF secret = 1234567890ABCDEF
To start the workers you can use the provided systemd files via systemctl start openqa-worker@1. This will start worker number one. You can start as many workers as you dare, you just need to supply different 'worker id' (number after @).
You can also run workers manually from command line.
install -d -m 0755 -o _openqa-worker /var/lib/openqa/pool/X sudo -u _openqa-worker /usr/share/openqa/script/worker --instance X
This will run a worker manually showing you debug output. If you haven’t installed 'os-autoinst' from packages make sure to pass --isotovideo option to point to the checkout dir where isotovideo is, not to /usr/lib! Otherwise it will have trouble finding its perl modules.
Where to now?
Setting up git support
Editing needles from web can optionally commit new or changed needles automatically to git. To do so, you need to enable git support by setting
[global] scm = git
in /etc/openqa/openqa.ini. Once you do so and restart the web interface, openQA will automatically commit new needles to the git repository.
You may want to add some description to automatic commits coming from the web UI. You can do so by setting your configuration in the repository (/var/lib/os-autoinst/needles/.git/config) to some reasonable defaults such as:
[user] email = firstname.lastname@example.org name = openQA web UI
To enable automatic pushing of the repo as well, you need to add the following to your openqa.ini:
[scm git] do_push = yes
Depending on your setup, you might need to generate and propagate ssh keys for user 'geekotest' to be able to push.
Referer settings to auto-mark important jobs
Automatic cleanup of old results (see GRU jobs) can sometimes render important tests useless. For example bug report with link to openQA job which no longer exists. Job can be manually marked as important to prevent quick cleanup or referer can be set so when job is accessed from particular web page (for example bugzilla), this job is automatically labeled as linked and treated as important.
List of recognized referers is space separated list configured in /etc/openqa/openqa.ini:
[global] recognized_referers = bugzilla.suse.com bugzilla.opensuse.org
Default behavior for all workers is to use the 'Qemu' backend and connect to 'http://localhost'. If you want to change some of those options, you can do so in /etc/openqa/workers.ini. For example to point the workers to the FQDN of your host (needed if test cases need to access files of the host) use the following setting:
[global] HOST = http://openqa.example.com
Once you got workers running they should show up in the admin section of openQA in the workers section as 'idle'. When you get so far, you have your own instance of openQA up and running and all that is left is to set up some tests.
Configuring remote workers
There are some additional requirements to get remote worker running. First is to ensure shared storage between openQA WebUI and workers. Directory /var/lib/openqa/share contains all required data and should be shared with read-write access across all nodes present in openQA cluster. This step is intentionally left on system administrator to choose proper shared storage for her specific needs.
Example of NFS configuration: NFS server is where openQA WebUI is running. Content of /etc/exports
NFS clients are where openQA workers are running. Run following command:
mount -t nfs openQA-webUI-host:/var/lib/openqa/share /var/lib/openqa/share
Configuring AMQP message emission
You can configure openQA to send events (new comments, tests finished, …) to an AMQP message bus. The messages consist of a topic and a body. The body contains json encoded info about the event. See amqp_infra.md for more info about the server and the message topic format. There you will find instructions how to configure the AMQP server as well.
To let openQA send messages to an AMQP message bus, first make sure that the perl-Mojo-RabbitMQ-Client RPM is installed. Then you will need to configure amqp in /etc/openqa/openqa.ini:
# Configuration for AMQP plugin [amqp] heartbeat_timeout = 60 reconnect_timeout = 5 # guest/guest is the default anonymous user/pass for RabbitMQ url = amqp://guest:guest@localhost:5672/ exchange = pubsub topic_prefix = suse
For a TLS connection use amqps:// and port 5671.
Configuring worker to use more than one openQA server
When there are multiple openQA web interfaces (openQA instances) available a worker can be configured to register and accept jobs from all of them.
/etc/openqa/client.conf must contain API keys and secrets to all instances
Shared storage from all instances must be properly mounted
In the /etc/openqa/workers.ini enter space-separated instance hosts and optionally configure where the shared storage is mounted. Example:
[global] HOSTS = openqa.opensuse.org openqa.fedora.fedoraproject.org [openqa.opensuse.org] SHARE_DIRECTORY = /var/lib/openqa/opensuse [openqa.fedoraproject.org] SHARE_DIRECTORY = /var/lib/openqa/fedora
Configuring SHARE_DIRECTORY is not a hard requirement. Worker will try following directories prior registering with openQA instance:
fail if none of above is available
Once worker registers to openQA instance it checks for available job and starts accepting websockets commands. Worker accepts jobs as they will come in, there is no priority, or other ordering, support at the moment. It is possible to mix local openQA instance with remote instances or use only remote instances.
If your network is slow or you experience long time to load needles you might want to consider to enable caching in your remote workers. To enable caching, /var/lib/openqa/cache must be created, and right permissions given to the 'geekotest' user. If you install openQA through the repositories, said directory will be created for you.
In the /etc/openqa/workers.ini
[global] HOST=http://webui CACHEDIRECTORY = $cache_location CACHELIMIT = 50 # This is currently noop. Cache supports limiting, but is not enabled. [http://webui] TESTPOOLSERVER = rsync://yourlocation/tests
Setup and run rsync server daemon on HOST machine, in /etc/rsyncd.conf should be:
gid = users read only = true use chroot = true transfer logging = true log format = %h %o %f %l %b log file = /var/log/rsyncd.log pid file = /var/run/rsyncd.pid slp refresh = 300 use slp = false #[Example] # path = /home/Example # comment = An Example # auth users = user # secrets file = /etc/rsyncd.secrets [tests] path = /var/lib/openqa/share/tests comment = OpenQA Test Distributions
systemctl start rsyncd.service systemctl enable rsyncd.service
This will allow the workers to download the assets from the webUI and use them locally. If TESTPOOLSERVER is set tests and needles will also be cached by the worker.
Auditing - tracking openQA changes
Auditing plugin enables openQA administrators to maintain overview about what is happening with the system. Plugin records what event was triggered by whom, when and what the request looked like. Actions done by openQA workers are tracked under user whose API keys are workers using.
Audit log is directly accessible from Admin menu.
Auditing, by default enabled, can be disabled by global configuration option in /etc/openqa/openqa.ini:
[global] audit_enabled = 0
The audit section of /etc/openqa/openqa.ini allows to exclude some events from logging using a space separated blacklist:
[audit] blacklist = job_grab job_done
List of events tracked by the auditing plugin:
iso_create iso_delete iso_cancel
job_create job_grab job_delete job_update_result job_done jobs_restart job_restart job_cancel job_duplicate
table_create table_update table_delete
user_new_comment user_update_comment user_delete_comment user_login
Some of these events are very common and may clutter audit database. For this reason job_grab and job_done events are blacklisted by default.
|Upgrading openQA does not automatically update /etc/openqa/openqa.ini. Review your configuration after upgrade.|
The openQA web interface can be started via MOJO_REVERSE_PROXY=1 morbo script/openqa in development mode.
/var/lib/openqa/ must be owned by root and contain several sub directories, most of which must be owned by the user that runs openQA (default 'geekotest'):
db contains the database lockfile
images is where the server stores test screenshots and thumbnails
share contains shared directories for remote workers, can be owned by root
share/factory contains test assets and temp directory, can be owned by root but sysadmin must create subdirs
share/factory/iso and share/factory/iso/fixed contain ISOs for tests
share/factory/hdd and share/factory/hdd/fixed contain hard disk images for tests
share/factory/repo and share/factory/repo/fixed contain repositories for tests
share/factory/other and share/factory/other/fixed contain miscellaneous test assets (e.g. kernels and initrds)
share/factory/tmp is used as a temporary directory (openQA will create it if it owns share/factory)
share/tests contains the tests themselves
testresults is where the server stores test logs and test-generated assets
Each of the asset directories (factory/iso, factory/hdd, factory/repo and factory/other) may contain a fixed/ subdirectory, and assets of the same type may be placed in that directory. Placing an asset in the fixed/ subdirectory indicates that it should not be deleted to save space: the GRU task which removes old assets when the size of all assets for a given job group is above a specified size will ignore assets in the fixed/ subdirectories.
It also contains several symlinks which are necessary due to various things moving around over the course of openQA’s development. All the symlinks can of course be owned by root:
script (symlink to /usr/share/openqa/script/)
tests (symlink to share/tests)
factory (symlink to share/factory)
It is always best to use the canonical locations, not the compatibility symlinks - so run scripts from /usr/share/openqa/script, not /var/lib/openqa/script.
You only need the asset directories for the asset types you will actually use, e.g. if none of your tests refer to openQA-stored repositories, you will need no factory/repo directory. The distribution packages may not create all asset directories, so make sure the ones you need are created if necessary. Packages will likewise usually not contain any tests; you must create your own tests, or use existing tests for some distribution or other piece of software.
The worker needs to own /var/lib/openqa/pool/$INSTANCE, e.g.
…. - add more if you have more CPUs/disks
You can also give the whole pool directory to the _openqa-worker user and let the workers create their own instance directories.
Tests fail quickly
Check the log files in /var/lib/openqa/testresults
KVM doesn’t work
make sure you have a machine with kvm support
make sure kvm_intel or kvm_amd modules are loaded
make sure you do have virtualization enabled in BIOS
make sure the '_openqa-worker' user can access /dev/kvm
make sure you are not already running other hypervisors such as VirtualBox
when running inside a vm make sure nested virtualization is enabled (pass nested=1 to your kvm module)
openid login times out
www.opensuse.org’s openid provider may have trouble with IPv6. openQA shows a message like this:
no_identity_server: Could not determine ID provider from URL.
To avoid that switch off IPv6 or add a special route that prevents the system from trying to use IPv6 with www.opensuse.org:
ip -6 r a to unreachable 2620:113:8044:66:130:57:66:6/128