Pytest should by default terminate on exceptions that inherit from BaseException and related changes to exception handling

[ceridwen]

At the moment if I use the signal module to catch SIGTERM and raise an exception, like the Python interpreter does by default for SIGINT and KeyboardInterrupt, sending SIGTERM will only cause one test to fail: pytest will catch the exception and continue running tests. This seems like the wrong behavior to me: there should be way to raise an exception that pytest won't continue on that isn't KeyboardInterrupt.

The reason I want to do this is that Jenkins, when it aborts a job, seems to send SIGTERM to its subprocesses, and when I abort a job, I almost always want the test clean-up code to run. I could call pytest.exit, except that then the signal handler is only useful when pytest itself is running, not when other Python code is running. I could write two signal handlers, one for pytest and one for everything else and then try to make sure that the correct signal handler is always set (note that the pytest signal handler also has to ensure that clean up is run outside pytest as well), or do stack introspection in the signal handler to try to determine if pytest is running, but both of those solutions look fragile and hard to get right; complexity is the last thing I want in my signal-handling code. I could raise KeyboardInterrupt, but KeyboardInterrupt when directly raised in code that pytest is executing causes pytest to terminate without running clean-up, unlike when SIGINT is sent to the Python interpreter, which defeats the purpose. This would also overload KeyboardInterrupt, which seems like bad style to me: KeyboardInterrupt should mean SIGINT, not other termination signals.

It seems to me the natural way to get Python to do clean-up correctly when handling SIGTERM is the way KeyboardInterrupt is handled by default, raise an exception. Thus, I'd propose that:

1. pytest should terminate on exceptions that inherit from BaseException, except for GeneratorExit. (This doesn't normally terminate the interpreter.) Of the built-in exceptions, this means only SystemExit and KeyboardInterrupt.

2. pytest.raises should remain able to catch SystemExit, KeyboardInterrupt, and other exceptions that inherit from BaseException and not terminate, so there's still a way to test expected instances of those exceptions, while allowing unexpected instances to cause pytest to exit.

3. pytest should allow clean-up code to run when KeyboardInterrupt is raised in Python code rather than via a signal for consistency in behavior.

(All my testing was done on pytest 3.6.0).

[asottile]

The hard (impossible) part here is this should be a failing test suite:

def test_that_fails():
    assert False

def test_oops_i_called_exit():
    exit(0)

given the proposal, I don't see a way to keep that from silently passing

[ceridwen]

I was imprecise in my terminology: pytest does catch KeyboardInterrupt, but immediately generates the report without failing the currently running test when that exception is thrown. I'm looking for that behavior rather than for pytest to exit without a report. I will point out that it's already possible to create that test suite though:

import os

def test_that_fails():
    assert False

def test_oops_i_called_exit():
    os._exit(1)

[ceridwen]

I've reworded some aspects of the proposal to make it clearer that I'm not suggesting that pytest not handle the exception at all when an exception inheriting from BaseException is raised, but rather interrupt the tests at that point and stop running the rest of the suite.

[anitahammer]

Not reacting to keyboard interrupts is a calling card of pytest, however it seems like it's over #12191