Skip to content

Why pytest?

Jump to bottom
Markus Gerstel edited this page Mar 3, 2018 · 4 revisions

The libtbx testing framework works, but has not seen much development over the past few years. As a result it misses many features that other modern testing frameworks offer.

A few of the limitations:

  • libtbx tests must be registered in a run_tests.py, which means that there are forgotten tests out there.
  • Writing tests requires a lot of boilerplace.
  • Together these two lead to a programming anti-pattern where libtbx runs one file with one run() method, which calls 20 test functions, each of which generally test more than one thing.
  • This makes the test output less useful (which test failed exactly, and what was it supposed to do?), especially when combined with basic assertion errors that do not provide much information.
  • libtbx tests are prone to false positives and false negatives: libtbx detects warnings although the test works fine if the output includes words such as 'error' (which is not a bad word), and libtbx does not distinguish between skipped and succeeded tests (as of 10/2017).
  • This means when tests are 'skipped' by commenting out one of the 20 test functions, by returning early, or removing them from run_tests we may only discover that years later.
  • There is no support for fixtures,
  • or test classes,
  • you can't just rerun only failed tests,
  • or within the same directory, ...

libtbx vs pytest

Here is a very basic example of a libtbx test, alongside the required entry in run_tests.py:

tests/TstFailure.py
def return_two():
  return 1 + 1

def return_three():
  return 4

def exercise_thing():
  assert return_two() + return_two() == 4
  print "OK"

def exercise_other_thing():
  assert return_two() + return_three() == 5
  print "OK"

if __name__ == '__main__':
  exercise_thing()
  exercise_other_thing()
run_tests.py
tst_list = [
  "$D/tests/TstFailure.py",
]

The same test rewritten as pytest would look like this:

tests/test_failure.py
def return_two():
  return 1 + 1

def return_three():
  return 4

def test_2_and_2_equals_4():
  assert return_two() + return_two() == 4

def test_2_and_3_equals_5():
  assert return_two() + return_three() == 5

Pytest automatically discovers these tests by looking into all files named test_*.py and picks all functions prefixed test_. This eliminates the need to edit another file. Further, we don't actually need to write the test function name twice, so we can be as verbose as we like.

Have a look at the outputs:

And this is what the pytest gives you:

Not only is the pytest output less verbose and more to the point - you also benefit from useful assertion information. Here are two further examples of rich assertions, and there are many more:

    def test_string_identity():
>     assert 'this long string contains one thing' == 'this long string contains another thing'
E     AssertionError: assert 'this long st...ins one thing' == 'this long str...another thing'
E       - this long string contains one thing
E       ?                            ^
E       + this long string contains another thing
E       ?                           ++ ^^ +

    def test_list_equality():
>     assert [ 1, 2, 3, 4, 5 ] == [ 1, 2, 3, 5 ]
E     assert [1, 2, 3, 4, 5] == [1, 2, 3, 5]
E       At index 3 diff: 4 != 5
E       Left contains more items, first extra item: 5
E       Use -v to get the full diff

Enabling pytests for a cctbx module

There is a compatibility layer in place so you can run libtbx tests with pytest and vice versa.

To enable libtbx to run pytests in your cctbx module you need to add pytest discovery to run_tests.py:

from libtbx.test_utils.pytest import discover
tst_list = [
 (...)
] + discover()

To enable pytest to run libtbx tests you need to add these lines to conftest.py in your module root:

from libtbx.test_utils.pytest import libtbx_collector
pytest_collect_file = libtbx_collector()

I would also recommend to add these lines to your libtbx_refresh.py to ensure that the pytest related modules are installed on all developer machines:

import libtbx.pkg_utils
libtbx.pkg_utils.require('mock', '>=2.0')
libtbx.pkg_utils.require('pytest', '>=3')

Now that you know why pytests are great and have it set up for your module, have a look at how you can use pytest in DIALS

Clone this wiki locally