zope-testrunner should exit with a non-zero exit code if it fails to find any tests

[mgedmin]

It's very easy to get the options wrong and end up with zope-testrunner not finding any tests. In this situation zope-testrunner exits with 0, which means wrappers such as tox or Travis CI consider the test run to be successfull and show a nice green checkmark, misleading the user into thinking everything's fine.

We should consider not finding any tests at all to be an error condition and exit with a non-zero status.

[cjwatson]

Can I suggest expanding this slightly? I think this should be done at the level of individual filters as well (e.g. -t or -m). This is because it's similarly easy to say something like zope.testrunner -t one-filter -t another-filter-with-a-typo and not notice that one of your filters has a typo, because after all you got some reasonable test output. IMO each filter should be required to match at least one test.

(If done correctly, this would also cover your request, because zope.testrunner defaults to something along the lines of -m . -t ..)

[cjwatson]

On the other hand, my suggestion does seem rather more complicated to implement now that I look at it, because zope.testrunner builds a single filtering predicate from all the given filters, and without rearranging that there's no way for it to check that each individual filter matched something. So don't let me derail this issue.

[icemac]

I have been bitten by this problem in zopefoundation/bobo#14: Everything seemed fine but no test ran. 😢