OpenQA Tests for Qubes OS

To install, clone this repo to /var/lib/openqa/tests/qubesos

Example command for posting new ISO for test:

openqa-client --host https://openqa.qubes-os.org isos post iso=Qubes-DVD-x86_64-20180224.iso DISTRI=qubesos VERSION=4.0 FLAVOR=install-iso ARCH=x86_64 BUILD=20180224 UEFI_DIRECT=1

Example command for starting tests of updates in current-testing:

openqa-client --host https://openqa.qubes-os.org isos post DISTRI=qubesos VERSION=4.0 FLAVOR=update ARCH=x86_64 BUILD=$(date +%Y%m%d%H) UEFI_DIRECT=1

Updates test on R4.1:

openqa-cli api isos -X post VERSION=4.1 BUILD=$(date +%Y%m%d%H) DISTRI=qubesos ARCH=x86_64 FLAVOR=update UPDATE=1 UPDATE_TEMPLATES=fedora-34-xfce GUIVM=1

Example command for starting tests for a template in templates-*-testing:

openqa-client --host https://openqa.qubes-os.org isos post DISTRI=qubesos VERSION=4.0 FLAVOR=templates ARCH=x86_64 BUILD=2020022706-4.0-fedora-31 UPDATE_TEMPLATES='fedora-31 fedora-31-minimal' TEST_TEMPLATES='fedora-31 fedora-31-minimal'

Job templates:

This is now edited via web UI. The templates.json file is git is a backup. Dump the templates: openqa-dump-templates --json | jq -S > templates.json Load the templates: openqa-load-templates templates.json

Variables:

• DISTRI - always qubesos
• VERSION - system version, currently not interpreted (but displayed in UI)
• FLAVOR - install-iso for installation tests, update for updates system tests
• ARCH - always x86_64
• BUILD - build number, not interpreted (but displayed in UI)
• UEFI_DIRECT - UEFI boots directly to Xen, no separate bootloader

Variables used in tests:

• INSTALL_TEMPLATES - space separated list of template groups (debian, whonix), or all; can include also specific template versions (debian-11) - they they will be installed/reinstalled from online repo after the initial install process
• USBVM - USB VM configuration: none (assert that already disabled by default), disable (disable explicitly), sys-usb (default), sys-net (combined with Net VM)
• KEYBOARD_LAYOUT - install with non-default keyboard layout; currently only us-colemak value is supported
• LOCALE - install with non-default locale; currently only en_DK.utf8 value is supported
• SYSTEM_TESTS - run system tests from those (space separated) modules, each module can be suffixed with ":TIMEOUT" (in seconds); tests are run using nose2
• VERBOSE_LIBVIRT - increase libvirt log level, including libxl logging; slows down the system and may lead to huge logs
• REPO_1 - repository name to install updates (dom0 and vm) from - in addition to standard repos
• KEY_1 - repository key short fingerprint to use for REPO_1
• SALT_SYSTEM_TESTS - run salt formula preparing for system tests (mostly install extra packages)
• UPDATE_TEMPLATES - update listed template packages as a whole (space separated list of templates to update)
• TEST_TEMPLATES - limit tests to listed templates only (space separated list of templates to test)
• DEFAULT_TEMPLATE - choose default template - on install jobs it's chosen in initial-setup, on update jobs all qubes that used default template are switched to this one
• UPDATE - install updates in dom0 and all templates
• QUBES_TEST_EXTRA_INCLUDE, QUBES_TEST_EXTRA_EXCLUDE - passed directly to the test environment
• TEST_GUI_INTERACTIVE - simple GUI interactive tests (start application from menu etc)
• TEST_WINDOWS_GUI_INTERACTIVE - simple GUI interactive tests for Windows with QWT installed
• WINDOWS_VERSION - create Windows VM of specific version and install QWT inside (argument as for qvm-create-windows-qube)
• QWT_DOWNLOAD - download specific QWT build (can be URL to iso or rpm file); otherwise qubes-windows-tools dom0 packages is installed from standard repositories
• GUIVM - create GUI VM (sys-gui)
• GUIVM_VNC - create GUI VM (sys-gui-vnc)
• GUIVM_GPU - create GUI VM (sys-gui-gpu)
• KEEP_SCREENLOCKER - do not disable xscreensaver (or whatever screenlocker is there)
• PARTITIONING - partition layout ('standard', 'xfs', 'btrfs', 'default', 'unencrypted'); on install - set via installer options, later - created manually on sdb, and all templates are migrated there
• HEADS - when set to 1, the test will handle Heads boot menu. Currently relevant only with BACKEND=generalhw. It's supposed to be sed in "machine" definition.
• RELEASE_UPGRADE_REPO - relevant for release-upgrade flavor, download scripts from specific URL, like: https://raw.githubusercontent.com/marmarek/qubes-dist-upgrade/<branch-name>
• WHONIXCHECK - run whonixcheck / systemcheck on all Whonix-related VMs (including templates)
• WHONIX_INTERACTIVE - run interactive tests on Whonix-related VMs (mostly anon-whonix)
• KERNEL_VERSION - which kernel flavor should be used - currently supported value is 'latest', which will switch to kernel-latest and kernel-latest-qubes-vm during update
• PIPEWIRE - when set to 1, VMs are set to use pipewire instead of pulseaudio, including switching native pulseaudio to pipewire-pulse.
• SUSPEND_MODE - what suspend mode should be used; default is S3, set to S0ix to use S0ix suspend
• INSTALL_OEM - do fully automated OEM installation
• INSTALL_OEM_STARTUP - start OEM installation, but still expect interactive prompts during installation
• HID - for generalhw tests, tell it whether USB hid is used (default) or PS2 one; in the latter case, sys-usb is not supposed to have input-proxy allowed by default

SecureDrop variables used in tests:

• SECUREDROP_INSTALL - when set to 1, SecureDrop Workstation is installed
• SECUREDROP_TEST - when set with a value, SecureDrop tests are run on top of an existing installation

Funding

TrenchBoot as Anti Evil Maid - UEFI boot mode support

This project is partially funded through NGI0 Core Fund, a fund established by NLnet with financial support from the European Commission's Next Generation Internet programme, under the aegis of DG Communications Networks, Content and Technology under grant agreement No 101092990.