Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: Extend development tricks with dynamic test override #1855

Merged
merged 1 commit into from
Nov 28, 2018
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 52 additions & 5 deletions docs/WritingTests.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -1134,11 +1134,14 @@ stop them outputting non-printable characters.


== Test Development tricks
=== Modifying setting of an existing test
=== Trigger new tests by modifying settings from existing test runs

There is no interface to modify existing tests but the clone_job.pl script
can be used to create a new job that adds, removes or changes
settings. This script is located at +/usr/share/openqa/script/+.
To trigger new tests with custom settings the command line client
+openqa-client+ can be used. To trigger new tests relying on all settings from
existing tests runs but modifying specific settings the +openqa-clone-job+
script can be used. Within the openQA repository the script is called
+clone_job.pl+ and is located at +/usr/share/openqa/script/+. This tool can
be used to create a new job that adds, removes or changes settings.

[source,sh]
--------------------------------------------------------------------------------
Expand All @@ -1163,7 +1166,7 @@ and the job will not appear as a job in any job group, e.g.:
clone_job.pl --from localhost 42 _GROUP=0
-------------

==== Backend variables for faster test execution
=== Backend variables for faster test execution

The +os-autoinst+ backend offers multiple test variables which are helpful for
test development. For example:
Expand Down Expand Up @@ -1248,3 +1251,47 @@ the clone source job or specifying the variable explicitly:
----
clone_job.pl --from https://openqa.opensuse.org --host localhost 24 TESTDEBUG=1 SKIPTO=consoletest-yast2_i
----

=== Defining a custom test schedule or custom test modules

Normally the test schedule, that is which test modules should be executed and
which order, is prescribed by the +main.pm+ file within the test distribution.
Additionally it is possible to exclude certain test modules from execution
using the os-autoinst test variables +INCLUDE_MODULES+ and +EXCLUDE_MODULES+ as
well as define a custom schedule using the test variable +SCHEDULE+.
Also test modules can be defined and overridden on-the-fly using a
downloadable asset.

For example one can schedule a job on a production instance with a custom
schedule consisting of two modules from the provided test distribution plus
one test module which is defined dynamically and downloaded as an asset from
an external trusted download domain:

----
openqa-clone-job --from https://openqa.opensuse.org --host https://openqa.opensuse.org 24 SCHEDULE=tests/boot/boot_to_desktop,tests/console/consoletest_setup,foo ASSET_1_URL=https://example.org/my/test/foo.pm
----

where "foo.pm" is a file in test module format, e.g. the following content:

----
use base 'consoletest';
use strict;
use testapi;

sub run {
select_console 'root-console';
assert_script_run 'foo';
}

sub post_run_hook {}
1;
----

For example this can be used in bug investigations or trying out new test
modules which are hard to test locally.
https://github.com/os-autoinst/os-autoinst/blob/master/doc/backend_vars.asciidoc
describes the +SCHEDULE+ parameter in details as well as the others. The
section "Asset handling" in the <<UsersGuide.asciidoc#usersguide,Users Guide>>
describes how downloadable assets can be specified. It is important to note
that the specified asset is only downloaded once. New versions must be
supplied as new, unambiguous download target file names.