Suites

Test Suites

Creating Test Suites

Test suites are collections of test methods defined at the package, module, class, or method level. To add a specific method to a test suite, use the @suite decorator:

    @suite('suite_1', 'suite_2')
    def test_method_1(self): 
        ...do stuff...

Or use @suite multiple times:

    @suite('suite_3', conditions=is_buildbot)
    @suite('suite_4')
    def test_method_2(self):
        pass

Typically, it's most useful to assign suites at the class-, module-, or package-level. This is done with a _suites variable.

For an individual class:

    class MyTestCase(TestCase):
      _suites = ['suite_1', 'suite_2']

Suites assigned at the class level will inherit to subclasses.

For a module or package, add the _suites variable at the top of the module or the package's __init__.py:

    _suites = ['suite_1', 'suite_2', ...]

These will also inherit down the source tree.

    class ATest(...):
        def test_1(self):
            pass

    class BTest(ATest):
        _suites = ['b_suite']

    class CTest(ATest):
        pass

In the example above, the suite will be applied to test_1, which is defined in ATest. Since CTest will use the same method, CTest.test_1 will also appear in b_suite. We'll fix this shortly.

Running Test Suites

This utility only works from the top-level directory.

To list:

testify yelp.tests --list-suites

To list tests in a suite:

testify yelp.tests --list-tests --include-suite=suite_name

Note that this can be used to list which tests will be run using a certain configuration. For example to show tests from a suite that would run on bucket 0 of 3 buckets:

testify yelp.tests --list-tests --bucket=0 --bucket-count=3

Test suites can be included or excluded. Note that exclusion takes priority over inclusion. Multiple --include-suite and --exclude-suite arguments can be passed to the suite script.

To run a suite:

testify yelp.tests --include-suite=suite_name

To exclude a suite:

testify yelp.tests --exclude-suite=exclude_suite_name

To run a suite, skipping sandbox-disabled tests:

testify yelp.tests --include-suite=photos --exclude-suite=sandbox-disabled

Bucketing your tests

When automating your test running with large projects, it's often convenient to run some of the concurrently to take advantage of multiple testing environments. For example, we use buildbot for continuous build and testing. Check out the bucket options:

testify yelp.tests --exclude-suite=sandbox-disabled --bucket-count=4 --bucket=0

This will run just ¼ of the tests. That ¼ is a deterministic set—meaning than you can also run buckets 1,2,3 and all your tests will get executed. The bucketing is based on hashing the test class and test method name, so you're assured that the same tests are run on the same bucket every time, as long as the bucket count stays constant.