SyTest is an integration testing system for Matrix homeserver implementations; primarily at present the Synapse server. It performs "black-box" testing, by starting up multiple home server instances and testing the interaction using regular HTTP interaction much as any other Matrix client would do. It can output test results either to an interactive terminal, or as a TAP-style test report, for continuous-integration test harnsses such as Jenkins.
SyTest requires a number of dependencies that are easiest installed from CPAN.
If these are not being installed directly into the system perl (as root), then you will first have to arrange that
cpancan install to somewhere writable as the non-root user you are running SyTest as, and that
perlcan see that.
Personally I arrange this by adding three lines to
export PERL5LIB=$HOME/lib/perl5 export PERL_MB_OPT=--install_base=$HOME export PERL_MM_OPT=INSTALL_BASE=$HOME
Alternatively, see https://metacpan.org/pod/local::lib#The-bootstrapping-technique
If you have edited your
.profile, don't forget to
sourceit again into your running shell.
If you have not run
cpanbefore, it will prompt for answers to several questions when it performs the initial setup. Running it once with no arguments will give you a chance to answer these questions. Most likely you can just let it configure "automatically":
$ cpan CPAN.pm requires configuration, but most of it can be done automatically. If you answer 'no' below, you will enter an interactive dialog for each configuration option instead. Would you like to configure as much as possible automatically? [yes] ... cpan> exit Lockfile removed.
sytestsource and install its dependencies:
git clone https://github.com/matrix-org/sytest cd sytest ./install-deps.pl cd ..
sytestis intended for testing the synapse home server implementation, it is likely you'll want to fetch the source of that as well. By default SyTest will expect to find it in a sibling directory called
pip install pynacl --user # Work around pynacl directory sort bug. git clone https://github.com/matrix-org/synapse cd synapse git checkout develop python setup.py develop --user python setup.py test cd ..
Synapse does not need to be installed, as SyTest will run it directly from its source code directory.
Additionally, a number of native dependencies are required. To install these dependencies on an Ubuntu/Debian-derived Linux distribution, run the following:
sudo apt install libpq-dev build-essential
Installing on OS X
Dependencies can be installed on OS X in the same manner, except that packages using NaCl / libsodium may fail. This can be worked around by:
Installing libsodium manually, eg.:
$ brew install libsodium
and confirm it is installed correctly and visible to pkg-config. It should give some configuration output, rather than an error:
$ pkg-config --libs libsodium -L/usr/local/Cellar/libsodium/1.0.8/lib -lsodium
Then force an install of Crypt::NaCl::Sodium:
$ cpan cpan> force install Crypt::NaCl::Sodium
You may also need to force install Shell::Guess, and manually install DBI before DBD::Pg, otherwise DBD::Pg will fail with:
No rule to make target '.../auto/DBI/Driver_xst.h'
Then run install-deps.pl as normal.
To run SyTest with its default settings, simply invoke the
script with no additional arguments:
cd sytest ./run-tests.pl
If the Synapse source is checked out in a different location, this can be set
./run-tests.pl --synapse-directory /home/synapse/synapse
If it is necessary to run the synapse server with a particular python
interpreter (for example, to run it within a virtualenv), this can be done
./run-tests.pl --python ../synapse/env/bin/python
If you're seeing the following error when trying to run against Synapse,
make sure you have your Synapse Python virtualenv(venv) activated:
$ ./run-tests.pl ... ** Error starting server-0 (on port 8800): python exited 1: /Users/myuser/.pyenv/versions/3.8.6/bin/python: Error while finding module specification for 'synapse.app.homeserver' (ModuleNotFoundError: No module named 'synapse') ❌ $ source ../synapse/env/bin/Activate (synapse) $ ./run-tests.pl ... ✅
To obtain greater visibility on why a particular test is failing, two
additional options can be passed to print extra information. The
--client-log flag (shortened to
-C) will print HTTP requests made and
--server-log flag (shortened to
-S) will print lines from the
Synapse server's standard error stream:
To run only a subset of tests in certain files, name the files as additional arguments:
To run synapse with a specific logging configuration, create a YAML file
suitable for dictConfig called
log.config (it can be copied from a running
synapse) and place it within the homeserver configuration directory
Sytest supports plugins. Plugins follow the same project structure as sytest and can be placed
plugins directory. They should contain the
lib/SyTest/Output directories, similar to the root of the sytest repository.
The path of the plugins directory can be overridden via the
SYTEST_PLUGINS environment variable.
Output implementations are supported in plugins.
See https://github.com/valkum/sytest_conduit for an example of a plugin.
For more information on developing SyTest itself (maintaining or writing new tests) see the DEVELOP.rst file.
Postgres Template Database
When testing with postgres SyTest will check if there is a database named sytest_template and will create the test databases using that as a template. This can be used to greatly reduce the time to create databases as they don't need to be created from scratch.
The easiest way to create the template database is to start a HS pointing at the database and stop it once the database has been created.