uncaught exception from testcase - exit without traceback

[e3krisztian]

Python 2.7.10 & 3.4.3

I have a failing test (in fact it is expected at them moment lacking some implementation), yet I get no traceback, nor do any remaining tests run - clean exit after printing a red E, which is quite embarrassing.

$ green lib
........................................E$

In fact I get the same clean exit from plain unittest

$ python -m unittest discover
........................................E$

Note: even a final new line was not printed!

unittest2 shows a relevant traceback and stops afterwards (no new tests are run).

Actually I am using testtools which depends on unittest2, so it works with that as well:

$ python -m testtools.run discover
Tests running...
======================================================================
ERROR: lib.tool.test_...
----------------------------------------------------------------------
Traceback (most recent call last):
...
  File "/usr/lib64/python3.4/argparse.py", line 1728, in parse_args
    args, argv = self.parse_known_args(args, namespace)
  File "/usr/lib64/python3.4/argparse.py", line 1767, in parse_known_args
    self.error(str(err))
  File "/usr/lib64/python3.4/argparse.py", line 2386, in error
    self.exit(2, _('%(prog)s: error: %(message)s\n') % args)
  File "/usr/lib64/python3.4/argparse.py", line 2373, in exit
    _sys.exit(status)
SystemExit: 2

nose runs properly, even capturing the uncaught second exception in unittest2 and continuing.

Unfortunately I could not create a minimal reproduction, yet (my attempts so far misteriously worked), but could pinpoint the place where it could be fixed.

Changing this line makes things work (though, this might not be the proper fix):

green/green/suite.py

Line 111 in 2a3b302

test(result)

            try:
                test(result)
            except:
                pass

[e3krisztian]

Reproduce with:

mkdir green-60
cd green-60
git clone --depth 1 --branch green-60 https://github.com/krisztianfekete/lib.git
cd lib
./test

[e3krisztian]

I have managed to reproduce it with a smaller code, see testing-cabal/testtools#144

Even if the original problem is elsewhere I still think green needs a fix too, as I get no help from it diagnosing the problem.

[CleanCut]

Ah, thank you! The smaller code helps. Sorry for the slow response, I have been swamped by life in general at the moment.

I agree, this should definitely be handled by green.

[CleanCut]

Okay, I finally got some time to sit down and look at this. Unfortunately, I can't reproduce your issue!

What am I missing?

I'm using this code (adapted from your post you linked to above):

# test.py
import unittest

class Test_testtools(unittest.TestCase):
    def test_1(self):
        pass

    def test_2(self):
        raise SystemExit(0)

    def test_3(self):
        pass

$ green test.py
.E.

Error in test.Test_testtools.test_2
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 331, in run
    testMethod()
  File "test.py", line 9, in test_2
    raise SystemExit(0)
SystemExit: 0

Ran 3 tests in 0.001s

FAILED (errors=1, passes=2)

[e3krisztian]

the first code fragment is the one - with testtools in it, as testtools causes the problem:

# test.py
import unittest
import testtools

class Test_testtools(testtools.TestCase):
    def test_1(self):
        pass

    def test_2(self):
        raise SystemExit(0)

    def test_3(self):
        pass

if __name__ == '__main__':
    unittest.main()

[CleanCut]

@krisztianfekete

It appears to be a conscious design choice that testtools will consider KeyboardInterrupt and SystemExit as being outside the scope of something that you want to test. In other words, they are letting those exceptions go through to the main python interpreter on purpose.

If you would like, you may review the relevant testtools code yourself (in their current master branch) at the following locations:

testtools/runtest.py:217-219 # Explanation of intent to re-raise certain exceptions
testtools/test_testcase.py:906-915 # Demonstrates they expect SystemExit to escape TestCase.run()
testtools/matchers/_exception.py:81-121 # Actual location that implements re-raising SystemExit and KeyboardInterrupt

That is contrary to the design used by unittest, unittest2, py.test, trial, nose, etc. -- at least as far as I am aware, all of the other tools that provide their own TestCase subclasses have designed them to catch all exceptions. Green relies on the internal exception-handling of those subclasses as part of the "Traditional" aspect of things (see green's feature list on the readme).

Since this is a design choice that testtools has made, for whatever reason, I'm not going to start battling them by overriding their behavior when tests are run by green. Going down that road would eventually lead to me battling other behavior of other tools and eventually creating ill will among their users and developers. Definitely not something that's worth it to me.

So, I think your options for handling it in your own code are:

1. Catch the SystemExit in your own test.
2. Ask testtools to catch SystemExit like most other TestCase implementations do.
3. Stop using testtools.TestCase

I recommend number 1. Quick, easy, and doesn't pick fights with anyone's design decisions.

Good luck!

[e3krisztian]

I am disappointed, maybe I have failed to convey my problem.

Let me try again:

I expect from a test runner

• to run tests
• provide feedback
• show information about failures

I have presented a test case, where all three above was compromised. This is clearly a problem.
Whatever caused the failure, green has failed as well.

The whatever in this case is a widely used test library, these are testtools' statistics on PyPI at the time of writing:

Downloads (All Versions):
10394 downloads in the last day
76699 downloads in the last week
240422 downloads in the last month

testtools unexpectedly from such a library breaks compatibility with its ancestors (L from SOLID).
green was not robust to handle the deviance.

Can I trust green (unittest, etc) anymore when third party libraries as widely used as testtools can cause them to fail silently? Zero exit status, no failure, no traceback!

When I found this problem it was due to a failing test: a library (argparse) unexpectedly for me raised SystemExit. I did not expect SystemExit.

---

"My options":

1. Do you seriously propose to catch all errors in every test case?
2. SystemExit prevents traceback and stops test execution testing-cabal/testtools#144
3. it is hard, as it is very nice to use with fixtures, but this option did occur to me

it looks quite funny (or not) when applied to the test in question and green ;-) :

1. Do not write such a test case or carefully read all the library sources/documentations that you use and be aware of all strange design decisions, test libraries, frameworks included
2. Ask green to gracefully handle failures in custom TestCase implementations (uncaught exception from testcase - exit without traceback #60)
3. Stop using green

Option 1 is DRY and assume and follow convention.
Option 2 is being nice.
I do not like option 3.

[CleanCut]

I agree that catching it in your one test won't prevent this situation from occurring again in other tests.

Catching the specific instance of SystemExit that is causing you problems would be a good, pragmatic solution to your current problem test in the short term.

Solving it in the long-term is always better.

It would be great for the testtools to conform to standard python TestCase behavior so that other python testing tools, including alternate test runners like green, will not have to each individually compensate for their behavior.

The testtools project appears to have established a good pattern of accepting pull requests. If you proposed a specific change in a pull request, maybe that would be enough to get them to change the behavior.

[e3krisztian]

Patching it in testtools will be hard and time consuming, it looks like a conscious decision and they even defended that decision over the years.

But even if it is patched green does not print stack trace next time. Would you accept the pull request that prints a stack trace?

[CleanCut]

I think your idea has merit. Global handling of any unanticipated exceptions would be an excellent way to handle any corner cases caused by poorly-coded testing frameworks (including my own, if I fall into the same trap). I will move over to the pull request and continue commenting there, instead of on this issue.

[CleanCut]

Version 1.10.0 was just released which contains the change that addresses this (and more...)