unified testing via toolz

[eriknw]

In the interest of development speed, the tests from toolz were basically just copied over to cytoolz. This is not easily sustainable. We would like all toolz-related tests to reside in toolz. Moreover, we would like a way to run the tests for toolz such as toolz.tests.run(), and to provide an optional toolz-like package on which to run the tests such as toolz.tests.run(cytoolz).

This issue does not cover testing CyToolz C API, which I regard as very low priority at the moment.

[eriknw]

As mentioned in #18, functions like assert_raises from nose.tools can be used, which will allow all failures to be displayed instead of just the first failure from a test suite. Development of unified testing could be a good time to make this switch.

[mrocklin]

Some testing systems, like py.test, might support these features even with normal assert syntax.

[mrocklin]

@minrk had suggestions about sharing test suites

1. cytoolz imports toolz, monkeypatches all toolz functions to cytoolz functions, and then calls toolz.test()
2. We make a separate toolz-test repository

[eriknw]

Monkeypatching FTW! This may be the best and cleanest use of monkey patching that I've seen. I like it.

How would toolz-test as a separate repository affect the workflow of development?

[tomprince]

pip install cytoolz
cd toolz/toolz/tests
cat <<EOF >conftest.py
import sys, cytoolz 
sys.modules['toolz'] = cytoolz
EOF
py.test .

[eriknw]

Slick.

I've been getting by with moving the tests from toolz to cytoolz as needed with the following script:

#!/bin/bash

for f in ../toolz/toolz/tests/test*py
do
    if [[ $f == *test_utils* ]]
    then
        continue
    fi
    newf=`echo $f | sed 's/...toolz.toolz/cytoolz/g'`
    sed -e 's/toolz/cytoolz/g' -e 's/itercytoolz/itertoolz/' \
        -e 's/dictcytoolz/dicttoolz/g' -e 's/funccytoolz/functoolz/g' \
        $f > $newf
    echo $f $newf
done

This has actually worked pretty well for me. Although this still results in tests being defined in two places, I have found that keeping cytoolz up to date with toolz--including complete tests--to be rather painless. The cytoolz-specific tests that ensure cytoolz is like toolz are especially convenient for keeping cytoolz up to date.

So, I don't know that we should pursue unified testing. To be fair, though, I think we should make it easy for anybody to move tests from toolz to cytoolz as I do with the script above, because I am not the only contributor.

Ideally, I would like the command $ make copytests to move the tests over. The goal of unified testing is to make it easy for anybody to run the latest tests from toolz for cytoolz. Adding a make target achieves this goal, imho, in a clean and straightforward way.

[llllllllll]

We could look at writing the tests in the structure of the CPython tests in modules like tests/test_operator.py that need to test both the c and python implementations.

[eriknw]

Closing. We have been getting by with $ make copytests to copy tests from toolz, adding cytoolz-specific tests in new files, and back-porting tests to toolz as necessary.