test/simplecyclic_test.py often fails

[felixdivo]

The fails can be seen on Linux and OSX. It seems like that test works rather non-deterministic.

[felixdivo]

test/back2back_test.py::Back2BackTestCase::test_timestamp might also be problematic. See here.

These tests might fail on the CI server but not on our own machines since the CI servers may be slower and are virtualized.

[felixdivo]

bb23d5b adds a better failure message.

[hardbyte]

Maybe a better approach would be to skip the tests are running on travis

[felixdivo]

Yes, that might be a good idea. Or to raise a skip exception if they fail instead of asserting. In that case, it still gets tested and reports an error of the class can't be found, the method signature changed or something like that.

EDIT:
So instead of using

self.assertTrue(90 < size < 110, '100 +/- 10 messages should have been transmitted. But queue contained {}'.format(size))

we could use

if not (90 < size < 110):
    if isTravisCI():
        raise unittest.SkipTest("what went wrong and the reason why this is allowed on Travis CI")
    else:
        self.fail('100 +/- 10 messages should have been transmitted. But queue contained {}'.format(size))

We could write a reusable method for this that looks like assertTrueButAllowFailureOnTravis(condition, failure_message=""). We should probably come up with a better name though. ;)

A simpler approach would be to annotate the test method @unittest.expectedFailure. But that would allow the test to fail even when run locally, which is not desired, right?.

[hardbyte]

How about using the unittest.skipIf decorator to always skip running these flaky tests if on travis?

[felixdivo]

unittest.skipIf would be the simplest and does the job I think.