Permalink
Fetching contributors…
Cannot retrieve contributors at this time
246 lines (166 sloc) 11 KB

openQA users guide

Introduction

This document provides additional information for use of the web interface or the REST API as well as administration information. For administrators it is recommend to have read the Installation Guide first to understand the structure of components as well as the configuration of an installed instance.

Use of the web interface

In general the web UI should be intuitive or self-explanatory. Look out for the little blue help icons and click them for detailed help on specific sections.

Some pages use queries to select what should be shown. The query parameters are generated on clickable links, for example starting from the index page or the group overview page clicking on single builds. On the query pages there can be UI elements to control the parameters, for example to look for more older builds or only show failed jobs or other settings. Additionally, the query parameters can be tweaked by hand if you want to provide a link to specific views.

/tests/overview - Customizable test overview page

The overview page is configurable by the filter box. Also, some additional query parameters can be provided which can be considered advanced or experimental. For example specifying no build will resolve the latest build which matches the other parameters specified. Specifying no group will show all jobs from all matching job groups. Also specifying multiple groups works, see the following example.

test overview page showing multiple groups
Figure 1. The openQA test overview page showing multiple groups at once. The URL query parameters specify the groupid parameter two times to resolve both the "opensuse" and "opensuse test" group.

Specifying multiple groups with no build will yield the latest build of the first group. This can be useful to have a static URL for bookmarking.

Description of test suites

Test suites can be described using API commands or the admin table for any operator using the web UI.

test suite description edit field
Figure 2. Entering a test suite description in the admin table using the web interface:

If a description is defined, the name of the test suite on the tests overview page shows up as a link. Clicking the link will show the description in a popup. The same syntax as for comments can be used, that is Markdown with custom extensions such as shortened links to ticket systems.

test suite description popup
Figure 3. popover in test overview with content as configured in the test suites database:

Review badges

Based on comments in the individual job results for each build a certificate icon is shown on the group overview page as well as the index page to indicate that every failure has been reviewed, e.g. a bug reference or a test issue reason is stated:

Review badges

Meaning of the different colors

  • The green icons shows up when there is no work to be done.

  • No icon is shown if at least one failure still need to be reviewed.

  • The black icon is shown if all review work has been done.

(To simplify, checking for false-negatives is not considered here.)

Show bug or label icon on overview if labeled gh#550

  • Show bug icon with URL if mentioned in test comments

  • Show bug or label icon on overview if labeled

For bugreferences write <bugtracker_shortname>#<bug_nr> in a comment, e.g. "bsc#1234", for generic labels use label:<keyword> where <keyword> can be any valid character up to the next whitespace, e.g. "false_positive". The keywords are not defined within openQA itself. A valid list of keywords should be decided upon within each project or environment of one openQA instance.

Example of a generic label
Figure 4. Example for a generic label
Example of a bug label
Figure 5. Example for bug label

Related issue: #10212

'Hint:' You can also write (or copy-paste) full links to bugs and issues. The links are automatically changed to the shortlinks (e.g. https://progress.opensuse.org/issues/11110 turns into poo#11110). Related issue: poo#11110

Also github pull requests and issues can be linked using the generic format `<marker>[#<project/repo>]#<id>`, e.g. gh#os-autoinst/openQA#1234, see gh#973

All issue references are stored within the internal database of openQA. The status can be updated using the /bugs API route for example using external tools.

Example for visualization of closed issue references
Figure 6. Example for visualization of closed issue references. Upside down icons in red visualize closed issues.

Build tagging

Tag builds with special comments on group overview

Based on comments on the group overview individual builds can be tagged. As 'build' by themselves do not own any data the job group is used to store this information. A tag has a build to link it to a build. It also has a type and an optional description. The type can later on be used to distinguish tag types.

The generic format for tags is

tag:<build_id>:<type>[:<description>], e.g. tag:1234:important:Beta1.

The more recent tag always wins.

A 'tag' icon is shown next to tagged builds together with the description on the group_overview page. The index page does not show tags by default to prevent a potential performance regression. Tags can be enabled on the index page using the corresponding option in the filter form at the bottom of the page.

Example of a tag coment and corresponding tagged build

Keeping important builds

As builds can now be tagged we come up with the convention that the 'important' type - the only one for now - is used to tag every job that corresponds to a build as 'important' and keep the logs for these jobs longer so that we can always refer to the attached data, e.g. for milestone builds, final releases, jobs for which long-lasting bug reports exist, etc.

Filtering test results and builds

At the top of the test results overview page is a form which allows filtering tests by result, architecture and TODO-status.

Filter form

There is also a similar form at the bottom of the index page which allows filtering builds by group and customizing the limits.

Highlighting job dependencies in 'All tests' table

When hovering over the branch icon after the test name children of the job will be highlighted blue and parents red. So far this only works for jobs displayed on the same page of the table.

highlighted child jobs

Use of the REST API

openQA includes a client script which - depending on the distribution - is packaged independantly if you just want to interface with an existing openQA instance without needing to install the full package. Call <openqa-folder>/script/client --help for help (openSUSE: openqa-client --help).

Basics are described in the Getting Started guide.

Triggering tests

Tests can be triggered over multiple ways, using clone_job.pl, jobs post, isos post as well as retriggering existing jobs or whole media over the web UI.

Cloning existing jobs - clone_job.pl

If one wants to recreate an existing job from any publically available openQA instance the script clone_job.pl can be used to copy the necessary settings and assets to another instance and schedule the test. For the test to be executed it has to be ensured that matching ressources can be found, for example a worker with matching WORKER_CLASS must be registered. More details on clone_job.pl can be found in Writing Tests.

Spawning single new jobs - jobs post

Single jobs can be spawned using the jobs post API route. All necessary settings on a job must be supplied in the API request. The "openQA client" has examples for this.

Spawning multiple jobs based on templates - isos post

The most common way of spawning jobs on production instances is using the isos post API route. Based on previously defined settings for media, job groups, machines and test suites jobs are triggered based on template matching. The Getting Started guide already mentioned examples. Additionally to the necessary template matching parameters more parameters can be specified which are forwarded to all triggered jobs. There are also special parameters which only have an influence on the way the triggering itself is done. These parameters all start with a leading underscore but are set as request parameters in the same way as the other parameters.

The following scheduling parameters exist
_NO_OBSOLETE

Do not obsolete jobs in older builds with same DISTRI and VERSION (as is the default behavior). With this option jobs which are currently pending, for example scheduled or running, are not cancelled when a new medium is triggered.

_DEPRIORITIZEBUILD

Setting this switch '1' will not immediately obsolete jobs of old builds but rather deprioritize them up to a configurable limit of priority.

_DEPRIORITIZE_LIMIT

The configurable limit of priority up to which jobs should be deprioritized. Needs _DEPRIORITIZEBUILD. Default 100.

_ONLY_OBSOLETE_SAME_BUILD

Only obsolete (or deprioritize) jobs for the same BUILD. This is useful for cases where a new build appearing doesn’t necessarily mean existing jobs for earlier builds with the same DISTRI and VERSION are no longer interesting, but you still want to be able to re-submit jobs for a build and have existing jobs for the exact same build obsoleted.

_GROUP

Job templates not matching the given group name are ignored. Does not affect obsoletion behavior, so you might want to combine with _NO_OBSOLETE.

_GROUP_ID

Same as _GROUP but allows to specify the group directly by ID.

Example for _DEPRIORITIZEBUILD and _DEPRIORITIZE_LIMIT.

openqa-client isos post ISO=my_iso.iso DISTRI=my_distri FLAVOR=sweet \
         ARCH=my_arch VERSION=42 BUILD=1234 \
         _DEPRIORITIZEBUILD=1 _DEPRIORITIZE_LIMIT=120 \

Where to now?

For test developers it is recommended to continue with the Test Developer Guide.