Pywikibot tests

The Pywikibot tests are based on the unittest framework, and are compatible with nose.

The tests package provides a function load_tests that supports the load tests protocol. The default ordering begins with tests of underlying components, then tests site and page semantics, and finishes with tests of the scripts and finally any tests which have not been inserted into the ordered list of tests.

A function collector also exists in the 'tests' package.

Running tests

All tests

The entire suite of tests may be run in the following ways from the root directory:

setup.py

python setup.py test

python setup.py nosetests --tests tests

pip install pytest-runner
python setup.py pytest

Module unittest

python -m unittest -v

nose

nosetests -v

pytest

py.test

tox

tox

Specific tests

Individual test components can be run using unittest, nosetests, or pwb

unittest

python -m unittest -v tests.api_tests tests.site_tests
python -m unittest -v tests.api_tests.TestParamInfo.test_init

nose

nosetests -v tests.api_tests tests.site_tests
python -m unittest -v tests.api_tests:TestParamInfo.test_init

pytest

py.test -s -v tests/api_tests.py tests/site_tests.py
py.test -s -v tests/api_tests.py::TestParamInfo::test_init

pwb

python pwb.py tests/api_tests -v
python pwb.py tests/site_tests -v
python pwb.py tests/api_tests -v TestParamInfo.test_init

env

PYWIKIBOT_TEST_MODULES=api,site python setup.py test

Travis CI

After changes are published into a github repository, tests may be run on travis-ci.org according to the configuration in .travis.yml .

When changes are merged into the main repository, they are replicated to https://github.com/wikimedia/pywikibot , and travis tests are run and published at https://travis-ci.org/wikimedia/pywikibot/builds . These tests use the Wikimedia global (SUL) account 'Pywikibot-test', which has a password securely stored in .travis.yml . See section env:global:secure.

Anyone can run these tests on travis-ci.org using their own github account, with code changes that have not been merged into the main repository. To do this:

create a github and travis-ci account
fork the main github repository https://github.com/wikimedia/pywikibot
enable builds from the travis profile page: https://travis-ci.org/profile
push changes into the forked git repository
watch the build at https://travis-ci.org/<username>/pywikibot/builds

Only travis-ci builds from the main repository can access the password for the Wikimedia account 'Pywikibot-test'. All tests which require a logged in user are skipped if the travis-ci build environment does not have a password.

To enable 'user' tests on travis-ci builds for a different repository, add a username and password to travis:

Go to https://travis-ci.org/<username>/pywikibot/settings
Add a new variable named PYWIKIBOT_USERNAME and a value of a valid Wikimedia SUL username
Add another variable named USER_PASSWORD, with the private password for the Wikimedia SUL username used in step 2. Check that this environment variable has "Display value in build logs" set to OFF, so the password does not leak into the build logs.
The next build should run tests that require a logged in user

If the username does not exist on one of the Travis build sites, user tests will not be run on that build site.

While passwords in travis-ci environment variables are not leaked in normal operations, you are responsible for your own passwords. If the variables contain single quotes it is necessary to surround them in double quotes (see also travis-ci #4350).

It is strongly recommended that an untrusted bot account is created for travis tests, using a password that is not shared with trusted accounts.

Appveyor CI

After changes are published into a github repository, tests may be run on a Microsoft Windows box provided by ci.appveyor.com according to the configuration in .appveyor.yml . To do this:

create a github and appveyor account
fork the main github repository
create a project in ci.appveyor.com
go to https://ci.appveyor.com/project/<username>/pywikibot/settings and enter the custom configuration .yml filename: .appveyor.yml
push changes into the forked git repository
watch the build at https://ci.appveyor.com/<username>/pywikibot/history

The 'user' tests are not yet enabled on appveyor builds.

CircleCI

After changes are published into a github repository, tests may be run on CircleCI Ubuntu servers.

create a github and circleci account
fork the main github repository
create a project in circleci.com
go to https://circleci.com/gh/<username>/pywikibot/edit#env-vars and add the following variables:
- PYWIKIBOT_NO_USER_CONFIG=2
- TOXENV=py27,py34
push changes into the forked git repository
watch the build at https://circleci.com/gh/<username>/pywikibot

PYWIKIBOT_NO_USER_CONFIG=2 is needed because 'python setup.py test' is run.

TOXENV=py27,py34 is a workaround because CircleCI runs 'tox', but there is a bug in the CircleCI default 'py26' implementation.

This approach does not include 'user' tests.

Environment variables

There are a set of 'edit failure' tests, which attempt to write to the wikis and should fail. If there is a bug in pywikibot or MediaWiki, these tests may actually perform a write operation.

These 'edit failure' tests are disabled by default. On Travis they are enabled by default on builds by any other github account except 'wikimedia'.

To disable 'edit failure' tests, set PYWIKIBOT_TEST_WRITE_FAIL=0

There are also several other 'write' tests which also attempt to perform write operations successfully. These will write to the wikis, and they should always only write to 'test' wikis.

These 'write' tests are disabled by default, and currently can not be run on travis or appveyor as they require interaction using a terminal. Also enabling them won't enable 'edit failure' tests.

To enable 'write' tests, set PYWIKIBOT_TEST_WRITE=1

Enabling only 'edit failure' tests or 'write' tests won't enable the other tests automatically.

Decorators

pywikibot's test suite, including Python's unittest module, provides decorators to modify the behaviour of the test cases.

@unittest.skipIf

Skip a test if the condition is true. Refer to unittest's documentation.

import unittest
[......]
@unittest.skipIf(check_if_fatal(), 'Something is not okay.')
def test_skipIf(self):

@unittest.skipUnless

Skip a test unless the condition is true. Refer to unittest's documentation.

import unittest
[......]
@unittest.skipUnless(check_if_true(), 'Something must happen.')
def test_skipUnless(self):

@tests.aspects.require_modules

Require that the given list of modules can be imported.

from tests.aspects import require_modules
[......]
@require_modules(['important1', 'musthave2'])
def test_require_modules(self):

@(unittest.)mock.patch

Replaces target with object specified in new. Refer to mock's documentation. This is especially useful in tests, where requests to third-parties should be avoided.

In Python 3, this is part of the built-in unittest module. But mock should be installed using pip for Python 2. Import whichever is available from the tests package.

from tests import patch


def fake_ping(url):
  return 'pong'
[......]
@patch('http_ping', side_effect=fake_ping)
def test_patch(self):
  self.assertEqual('pong', http_ping())

Contributing tests

Test modules should be named according to the pywikibot that is being tested. e.g. the module pywikibot.page is tested by tests.page_tests.

New test classes should be added to the existing test modules unless it tests a new component of pywikibot.

All test classes must be a subclass of tests.aspects.TestCase, which uses a metaclass to dynamically check the test can be run on a specified site, or run a test on multiple sites.

Test sites

If a test depends on a specific site, add class attributes 'family' and code'.

family = 'wikipedia'
code = 'en'

Once declared, the Site object can be accessed at self.site.

If a test requires multiple specific sites, add a class attribute 'sites'.

sites = {
    'enwiki': {
        'family': 'wikipedia',
        'code': 'en',
    },
    'itwikt': {
        'family': 'wiktionary',
        'code': 'it',
    }
}

To obtain the Site object, call self.get_site with the key given to the site.

self.get_site('itwikt')

For tests which require network access to a website which is not an APISite, the class attribute 'sites' may include a hostname.

sites = {
    'wdq':
        'hostname': 'wdq.wmflabs.org',
    }
}

Other class attributes

net = False : test class does not use a site
dry = True : test class can use a fake site object
cached = True: test class may aggressively cache API responses
cacheinfo = True: report cache hits and misses on tearDown
user = True : test class needs to login to site
sysop = True : test class needs to login to site as a sysop
write = True : test class needs to write to a site

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

README.rst

README.rst

Pywikibot tests

Running tests

All tests

setup.py

Module unittest

nose

pytest

tox

Specific tests

unittest

nose

pytest

pwb

env

Travis CI

Appveyor CI

CircleCI

Environment variables

Decorators

@unittest.skipIf

@unittest.skipUnless

@tests.aspects.require_modules

@(unittest.)mock.patch

Contributing tests

Test sites

Other class attributes

Files

README.rst

Latest commit

History

README.rst

File metadata and controls

Pywikibot tests

Running tests

All tests

setup.py

Module unittest

nose

pytest

tox

Specific tests

unittest

nose

pytest

pwb

env

Travis CI

Appveyor CI

CircleCI

Environment variables

Decorators

@unittest.skipIf

@unittest.skipUnless

@tests.aspects.require_modules

@(unittest.)mock.patch

Contributing tests

Test sites

Other class attributes