How should we handle SystemExit?

[aarchiba]

PR #2189 demonstrates that we handle SystemExit (and GeneratorExit) in slightly inconsistent ways - if raised from inside a strategy it is handled like a normal exception; but if raised from within a test function it is captured and treated like KeyboardInterrupt in some ways, ensuring for example that it doesn't trigger "Flaky" if raised some of the time but not others. What is the correct behaviour?

[aarchiba]

I think it makes sense to treat SystemExit and GeneratorExit like we do normal exceptions when raised from both strategies and test functions. This requires some changes, as when they arise from test functions they are currently handled more like KeyboardInterrupt, in particular not triggering Flaky if raised some times and not others.

[DRMacIver]

I feel like SystemExit and GeneratorExit are slightly different things: Someone calling sys.exit probably meant it, while a raised GeneratorExit is probably a bug (because neither tests nor strategies are coroutines - they might wrap coroutines, but that wrapper should be handling the exit).

This doesn't necessarily mean that they should be handled differently though.

I do think they're fundamentally different from KeyboardInterrupt, which can basically be triggered at any time and doesn't mean much, while these should happen predictably.

My inclination would be that SystemExit and KeyboardInterrupt should always be propagated but that we should probably treat GeneratorExit like any other exception. I'm not strongly wedded to this view though, and might be missing something about GeneratorExit.

[aarchiba]

Well, one way GeneratorExit might appear is, if I recall correctly, nose uses (used?) generators to implement parametrized tests - you'd write a generator that yielded sets of arguments. If your test runner bailed out without running all the tests, that generator would receive a GeneratorExit as it exited "out the side" without finishing.

But my reasoning was just that: both GeneratorExit and SystemExit are emitted deterministically by code, so checking for flakiness in their emission makes sense.

The remaining question is whether a function under test should be able to actually exit from hypothesis testing with sys.exit(), but already pytest intercepts and reports that more or less like a normal exception (and does not exit to the OS when it appears).

[Zac-HD]

nose uses (used?) generators to implement parametrized tests

@given raises an error if the wrapped function returns anything but None, so we don't need to worry about this at least!

IMO "SystemExit ought to be deterministic and pytest treats it as a normal exception" is enough to justify us doing the same.

[Zac-HD]

In principle this can be solved by adding GeneratorExit and SystemExit to the list here:

hypothesis/hypothesis-python/src/hypothesis/core.py

Lines 468 to 477 in 6466600

	def failure_exceptions_to_catch():
	"""Return a tuple of exceptions meaning 'this test has failed', to catch.
	This is intended to cover most common test runners; if you would
	like another to be added please open an issue or pull request.
	"""
	exceptions = [Exception]
	if "_pytest" in sys.modules: # pragma: no branch
	exceptions.append(sys.modules["_pytest"].outcomes.Failed)
	return tuple(exceptions)

And then writing a test that e.g.

@given(st.integers())
def test(x):
    if x > 100:
        raise SystemExit

prints output including Falsifying example: and x=101... which at the moment it doesn't. Unclear why so this isn't quite a "good first issue", but probably suitable for a first contribution to Hypothesis.