Subtest failure not handled

[ShadowLNC]

If a failure occurs within a subtest, Hypothesis doesn't seem to "catch" it.

Here's some example code:

from unittest import TestCase
from hypothesis import given, strategies as st

class Testing(TestCase):
    @given(myval=st.integers())
    def test1(self, **kwargs):
        self.assertTrue(False)  # Example AssertionError

    @given(myval=st.integers())
    def test2(self, **kwargs):
        with self.subTest(name="subtest"):
            self.assertTrue(False)  # Another example.

test1 fails and is correctly falsified (a single failure is printed, albeit with the full stack trace including Hypothesis's internal calls - I am unsure if this is intended, but it's irrelevant to this bug report).

test2, using a subtest, fails 100 times (100 different values tested), each one printing the final call of the stack trace (i.e. self.assertTrue(False) without the internal calls). No falsifying example is printed, nor is any indicator of the value(s) used. (The identifier (name='subtest') is shown in the failure line, as expected.)

I'm running Python 3.6.4 on Windows 10 Pro 64-bit, version 1709, build 16299.192.

Virtual Environment Packages (Django, Hypothesis, and dependencies):

attrs==17.4.0
coverage==4.4.2
Django==2.0.1
hypothesis==3.44.16
pytz==2017.3

[Zac-HD]

Unfortunately this is the expected behaviour - subtests clash very badly with Hypothesis, because a new subtest is added every time Hypothesis tries another example, and this "leaks" out of Hypothesis' control to the test runner. We will probably never support this as a feature, but I'll add an explicit warning about it to the docs - #1072.

I'd also like to detect this and issue an explicit warning. While this seems similar to #991, the very nature of subTest in acting at runtime instead of definition time makes it very hard to detect - I'm leaving the issue open for now but don't expect to make any progress 😢

[DRMacIver]

We will probably never support this as a feature

It's actually not too hard to support it. All we'd really need is a custom unittest subclass (which we could even install automatically, thought that might be a bit fraught) which adds some hooks and overrides so that Hypothesis clears the subtests at the beginning of each run and turns any subtests failing into a test failure.

That being said, it's probably not going to get done any time soon unless someone is interested in picking up the work. Our unittest support is a bit basic in large part because I don't think any of us actually use unittest.

[Zac-HD]

Hmm. What about monkey-patching subTest when @given is used to wrap another method on some TestCase subclass? We could replace it with an implementation that checks whether it's being called within a Hypothesis test, and if so avoids adding the subtest to output. Off the top of my head, "checking if we're currently in a hypothesis test" is the only part I couldn't do easily 😄

(update: nope; at the time of wrapping methods haven't been bound to classes)