Inexplicable test failures on 3.13.0+

[mithrandi]

I have a test that looks something like this:

@given(s.data(), fs.finapps(store))
@settings(suppress_health_check=[HealthCheck.too_slow])
def test(data, finapp):
    pass

On Hypothesis 3.12.0 this works fine. On Hypothesis 3.13.0, every example is rejected, eventually hitting the timeout (if I turn up the verbosity, I see "Run complete after 288 examples (0 valid) and 0 shrinks").

However, if I change it to this, which seems like it should be identical, then the test works fine on both versions of Hypothesis:

@given(s.data())
@settings(suppress_health_check=[HealthCheck.too_slow])
def test(data):
    finapp = data.draw(fs.finapps(store))

Unfortunately this codebase is proprietary, and fs.finapps is a pretty complex strategy, so I'm not sure exactly where to start tracking this down, and I haven't had any luck constructing a minimal reproducer yet.

[Zac-HD]

We certainly consider it a bug when public APIs stop working, but I don't think we can help much without knowing more about the strategy in question or a reproducible example - minimal is preferred but not absolutely required.

I suspect that @DRMacIver might also be able to help out 'for a modest fee' if that would help with the legal side of things.

[mithrandi]

Success! Once I realised that most of the code involved in my test was actually irrelevant, I was able to cut it down to something pretty small. Here's a self-contained example:

from hypothesis import strategies as s, given

def strat():
    return s.builds(dict, one=strat_one())

@s.composite
def strat_one(draw):
    return draw(s.builds(dict, val=s.builds(dict, two=strat_two())))

@s.composite
def strat_two(draw):
    return draw(s.builds(dict, some_text=s.text(min_size=1)))

@given(strat())
def test(v):
    pass

test()

This passes instantly on Hypothesis 3.12.0, and hits the timeout on 3.13.0 and 3.14.0 with zero examples found.

[marcinpohl]

This is with Python 3.6.2 from OSX Brew, and Hypothesis 3.14.1

Small lists get generated fine, just as before:

In [44]: listofstrings = lists(elements=text(), min_size=3, max_size=30)

In [45]: listofstrings.example()
Out[45]:
["\x10\U0004e195\U0004921d\U00104e30\x0f\U00086310\x06')+\x19\U001054ee ",
 "\U00060c28+.\x0e\U0004a174'\U00070347",
 '\U001083c1\n',
 '',
 '\U000a87d2',
 '\x12\x19',
 '\U00012c38\x1e',
 '',
 '\U0001c5c9!\U000c5b7c\U000e6225\x06($\U000f5ac7\U0006432c\U00019fef.',
 '\U000c7855\U00108895\U00050d5b\U000f3ff4\U00105472\U000ff837\U000dc88a\U000c77d8\U000f2ac1\U0007397f',
 '\x16\x14\U000c9462',
 '\U00091529\U00098d1b\x00',
 '\x1d',
 '\x0b(',
 '\x1a\U0003ced0\U0008fe78\U0009d0cc\U00088ca4\U000e4471\x18\U000f0ebb罵',
 '\U000664a7\U00013d70\U00071745)\U000b81f8"\U000dd610\U00039c7f\x05\x12',
 '\x01\ue608\x19',
 '']

Bigger lists (the breaking point is about 250-270) generate a list of empty strings:
In [48]: listofstrings = lists(elements=text(), min_size=300, max_size=900)
(I'm not pasting the output, unless you want to know what 900 lines of empty strings look like).

Also, the result between 300..900 elements keeps generating 900 element list, on multiple invocations of .example() which is odd.

When producing same size bigger lists but with unique=True, it tries for a while, hits a timeout, and produces this:

In [52]: listofstrings = lists(elements=text(), min_size=300, max_size=900, unique=True)

In [53]: listofstrings.example()
---------------------------------------------------------------------------
Unsatisfiable                             Traceback (most recent call last)
~/.virtualenvs/pybrew362/lib/python3.6/site-packages/hypothesis/searchstrategy/strategies.py in example(self, random)
    138                     database=None,
--> 139                     verbosity=Verbosity.quiet,
    140                 )

~/.virtualenvs/pybrew362/lib/python3.6/site-packages/hypothesis/core.py in find(specifier, condition, settings, random, database_key)
    799                 get_pretty_function_description(condition),
--> 800                 runner.valid_examples,))
    801

Unsatisfiable: Unable to satisfy assumptions of lambda x: True. Only 0 examples considered satisfied assumptions

During handling of the above exception, another exception occurred:

NoExamples                                Traceback (most recent call last)
<ipython-input-53-e890e750aa1d> in <module>()
----> 1 listofstrings.example()

~/.virtualenvs/pybrew362/lib/python3.6/site-packages/hypothesis/searchstrategy/strategies.py in example(self, random)
    142         except (NoSuchExample, Unsatisfiable):
    143             raise NoExamples(
--> 144                 u'Could not find any valid examples in 100 tries'
    145             )
    146

NoExamples: Could not find any valid examples in 100 tries

[DRMacIver]

@marcinpohl To some extent, being bad at large examples is a deliberate design choice. Hypothesis operates on a fixed size buffer for its example representation, and generally isn't going to work very well for any strategy that doesn't have any small examples - such things are both harder to find good examples in and less likely to work because of the comparatively slow tests.

That being said, we can definitely do better in this case. Pull request incoming.