Black-box integration testing for Matrix homeservers
Switch branches/tags
SYT-8-login.recaptcha add_push_rule_enabled_test anoa/as_config_files anoa/dendrite_print_default anoa/homeserver_specific_tests anoa/refactor_critical anoa/user_username babolivier/dendrite-log-config babolivier/messages-from-query babolivier/remove-critical bugs/SYN-391 dbkr/can_see_invite_rejection dbkr/change_password_tests dbkr/e2e_backup_versions_are_numbers dbkr/e2e_backups dbkr/federated_guest_join dbkr/get_pushers dbkr/idempotent_registration dbkr/join_group dbkr/non_ascii_passwords dbkr/notifications_api dbkr/publicity_broken dbkr/push_badges dbkr/register_idempotent_with_username dbkr/registration_remember_params dbkr/room_list_spider dbkr/test_invites_pushed dendrite develop erikj/SYN-589 erikj/account_deactivate erikj/add_state_to_timeline erikj/alias_deletion erikj/alias_room_id erikj/appservice_stream erikj/backfill_filter erikj/backfill_fix erikj/context_fix erikj/context erikj/create_room_3pid_invite erikj/delete_alias erikj/dendrite_naffka erikj/device_flake erikj/device_list_stream erikj/erikj/remove_events_stream_2 erikj/event_creator_worker_sytest erikj/events_creator erikj/events_order erikj/faster_get_joined erikj/fed_reader erikj/file_api2 erikj/filtered_presence erikj/fix_flake_lazy erikj/fix_pg_dendrite_config erikj/fix_presence_initial erikj/fix_push erikj/fix_race_backfill erikj/fix_signed_3pid erikj/fix_syn_606 erikj/fixup_forward_pagination_test erikj/flakey_push_test erikj/forget erikj/frontend_proxy erikj/frozen_event erikj/group_server erikj/groups_merged erikj/guest_event_tightloop erikj/haproxy_map erikj/ignore_user erikj/jenkins_reuse_venv erikj/joined_rooms erikj/leo_leaving erikj/logout erikj/media_from_cache erikj/member erikj/no_with_events erikj/order erikj/paginate_sync erikj/pg_params erikj/pg_template erikj/presence_fix erikj/presence erikj/public_room_fix erikj/public_room_paginate erikj/public_rooms_splitout erikj/purge_history erikj/push_disable_enable erikj/push_synced erikj/push_test erikj/quarantine_media erikj/read_markers erikj/real_user_peeking erikj/receipts_synced erikj/refactor_deletions erikj/reject_invite_empty_room erikj/reject_invite_federation erikj/rejoin_device_lists erikj/remote_invited_metadata erikj/remote_public_rooms erikj/remove_events_stream erikj/repl_tcp_client erikj/repl_tcp erikj/report_event erikj/room_create erikj/room_initial_sync_race_fix erikj/room_list erikj/search_fix erikj/share_room_user_dir erikj/shared_secret erikj/shutdown_room erikj/sign_fed erikj/sign_join erikj/spider erikj/split_out_fed_txn erikj/split_out_user_dir erikj/state_ids_api erikj/syn-611 erikj/syn-627 erikj/sync_flake_tests erikj/sync erikj/test_push_names erikj/test_redactions erikj/timeout erikj/typing_fed_timeout erikj/user_dir erikj/visibility erikj/3pid_guest_config erikj/401_async hawkowl/coverage hawkowl/frozendicts-var hawkowl/nonascii-pushrules hawkowl/py3-3 hawkowl/setup-dependencies homeserver-base-class hs/media-limits hs/upload-limits kegan/idempotent-requests kegan/join-with-custom-content leonerd/excise-LMU leonerd/federation-server-path-segments leonerd/flaky-tests leonerd/neater-repeat-loops markjh/archived_flag markjh/async_commit markjh/bcrypt_rounds markjh/bind_host markjh/branch_coverage markjh/cached_sync markjh/change_action markjh/coverage markjh/cpanfile markjh/deep_boolean markjh/default_rules markjh/delete_pusher markjh/dendron markjh/direct_to_device_federation_sync markjh/direct_to_device_federation markjh/direct_to_device_wildcard markjh/direct_to_device markjh/disable_jenkins_synchrotron markjh/enable_jenkins_synchrotron markjh/filter_inline markjh/fix-anonymous-access-tests markjh/fix_ban_propagation markjh/fix_race markjh/fix_wait_for_sync markjh/fixed markjh/flaky_presence markjh/flaky_test markjh/get_enabled markjh/ghost_displaynames markjh/guest_access markjh/guest_auth markjh/history_for_rooms_that_have_been_left markjh/idempotent_state markjh/idempotent markjh/ignore_user_synced markjh/initial_sync_archived_flag markjh/initialsync_room_sub markjh/invalid_filters markjh/invite_state markjh/jenkins2 markjh/joined_guest_access markjh/kill_listeners markjh/killoldlisteners markjh/less_magic markjh/license markjh/listeners markjh/logging markjh/login markjh/media_repository markjh/more_push_tests markjh/new_user markjh/open_id markjh/poll_synapse_http markjh/presence_filtering markjh/push_rules_sync markjh/push_rules markjh/pushrule_torture markjh/relative_push_rules markjh/rename_pud_to_account_data markjh/room_tags markjh/search_context markjh/set_avatar_url markjh/setup_timeout markjh/split_pusher markjh/split_pusherI markjh/syn-482-room-initialsync markjh/synchrotron markjh/timeouts markjh/twisted_debugging markjh/typing_sub markjh/unused_dendron_flags markjh/userid_in_uri markjh/v2_sync_api markjh/v2_sync_archived markjh/v2_sync_joining markjh/v2_sync_prev_content markjh/v2_sync_receipts markjh/v2_sync_typing markjh/wait_for_sync markjh/worker_config markjh/3pid_login master matthew/auto-displayname matthew/disable-ll-on-incr-syncs matthew/encrypt-for-invited-users matthew/filter_members matthew/fix-filename-escaping matthew/fix-self-ll matthew/fix_overzealous_ll_state matthew/heroes-for-avatars matthew/lazy_load_apis matthew/lazy_load_messages_api matthew/lazy_load_yourself matthew/matthew/lazy_load_yourself matthew/members_at matthew/more-gappy-tests matthew/remove_redundant_lazy_members matthew/room_summary matthew/sync_deleted_devices neilj/create_support_user neilj/fix_01register_user_type paul/SYN-680 paul/SYN-738 paul/SYT-47 paul/federation paul/get-state-whole-event paul/haproxy paul/homeserver-base-class paul/issue-1382 paul/matrix-get-events paul/no-premature-fixture-start paul/room-joined-members paul/rt.perl-128774 paul/ruma paul/third-party-lookup paul/tiny-fixes-to-sytest paul/unconditionalise-dendron-components paul/wait-for-exit rav/create_room_synced_returns_sync rav/docker_install_jq rav/event_context_shiz_release rav/failure_messages_in_junit rav/fix_federation_errors rav/fix_testscript_rsync rav/fix_tox rav/general_muppetry rav/improve_install_pain rav/no_powerlevel_takeover rav/pg_test rav/respect_event_format_in_filter rav/state_reset_test rav/test_fail readme_os_x release-v0.12.1 release-v0.34.0 uhoreg/e2e_cross-signing unread_notification_count
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
docker
docs
jenkins
keys
lib/SyTest
tests
.dir-locals.el
.dockerignore
.gitignore
CONTRIBUTING.rst
DEVELOP.rst
LICENSE
README.rst
STYLE-GUIDE.rst
cpanfile
install-deps.pl
run-tests.pl
tap-to-junit-xml.pl

README.rst

SyTest

SyTest is an integration testing system for Matrix homeserver implementations; primarily at present the Synapse server. It performs "black-box" testing, by starting up multiple home server instances and testing the interaction using regular HTTP interaction much as any other Matrix client would do. It can output test results either to an interactive terminal, or as a TAP-style test report, for continuous-integration test harnsses such as Jenkins.

Installing

SyTest requires a number of dependencies that are easiest installed from CPAN.

  1. If these are not being installed directly into the system perl (as root), then you will first have to arrange that cpan can install to somewhere writable as the non-root user you are running SyTest as, and that perl can see that.

    Personally I arrange this by adding three lines to .profile:

    export PERL5LIB=$HOME/lib/perl5
    export PERL_MB_OPT=--install_base=$HOME
    export PERL_MM_OPT=INSTALL_BASE=$HOME
    

    Alternatively, see https://metacpan.org/pod/local::lib#The-bootstrapping-technique

    If you have edited your .profile, don't forget to source it again into your running shell.

  2. If you have not run cpan before, it will prompt for answers to several questions when it performs the initial setup. Running it once with no arguments will give you a chance to answer these questions. Most likely you can just let it configure "automatically":

    $ cpan
    
    CPAN.pm requires configuration, but most of it can be done automatically.
    If you answer 'no' below, you will enter an interactive dialog for each
    configuration option instead.
    
    Would you like to configure as much as possible automatically? [yes]
    
    ...
    
    cpan[1]> exit
    Lockfile removed.
    
  3. Fetch the sytest source and install its dependencies:

    git clone https://github.com/matrix-org/sytest
    cd sytest
    ./install-deps.pl
    cd ..
    
  4. As sytest is intended for testing the synapse home server implementation, it is likely you'll want to fetch the source of that as well. By default SyTest will expect to find it in a sibling directory called synapse:

    pip install pynacl --user # Work around pynacl directory sort bug.
    git clone https://github.com/matrix-org/synapse
    cd synapse
    git checkout develop
    python setup.py develop --user
    python setup.py test
    cd ..
    

    Synapse does not need to be installed, as SyTest will run it directly from its source code directory.

Installing on OS X

Dependencies can be installed on OS X in the same manner, except that packages using NaCl / libsodium may fail. This can be worked around by:

Installing libsodium manually, eg.:

$ brew install libsodium

and confirm it is installed correctly and visible to pkg-config. It should give some configuration output, rather than an error:

$ pkg-config --libs libsodium
-L/usr/local/Cellar/libsodium/1.0.8/lib -lsodium

Then force an install of Crypt::NaCl::Sodium:

$ cpan
cpan> force install Crypt::NaCl::Sodium

You may also need to force install Shell::Guess, and manually install DBI before DBD::Pg, otherwise DBD::Pg will fail with:

No rule to make target '.../auto/DBI/Driver_xst.h'

Then run install-deps.pl as normal.

Running

To run SyTest with its default settings, simply invoke the run-tests.pl script with no additional arguments:

cd sytest
./run-tests.pl

If the Synapse source is checked out in a different location, this can be set with --synapse-directory:

./run-tests.pl --synapse-directory /home/synapse/synapse

If it is necessary to run the synapse server with a particular python interpreter (for example, to run it within a virtualenv), this can be done using --python:

./run-tests.pl --python ../synapse/env/bin/python

To obtain greater visibility on why a particular test is failing, two additional options can be passed to print extra information. The --client-log flag (shortened to -C) will print HTTP requests made and responses received:

./run-tests.pl -C

The --server-log flag (shortened to -S) will print lines from the Synapse server's standard error stream:

./run-tests.pl -S

To run only a subset of tests in certain files, name the files as additional arguments:

./run-tests.pl tests/20profile-events.pl

To run synapse with a specific logging configuration, create a YAML file suitable for dictConfig called log.config (it can be copied from a running synapse) and place it within the homeserver configuration directory (localhost-<port>).

Developing

For more information on developing SyTest itself (maintaining or writing new tests) see the DEVELOP.rst file.

Postgres Template Database

When testing with postgres SyTest will check if there is a database named sytest_template and will create the test databases using that as a template. This can be used to greatly reduce the time to create databases as they don't need to be created from scratch.

The easiest way to create the template database is to start a HS pointing at the database and stop it once the database has been created.