Odd regression in sets in 3.16.1

[mithrandi]

In 3.17.0, the following runs fine:

from hypothesis import given, strategies as s

@given(s.sets(s.sampled_from([0])))
def test(ign): pass
test()

In 3.18.0, there is a failing health check:

FailedHealthCheck: It looks like your strategy is filtering out a lot of data. Health check found 50 filtered examples but only 9 good ones. This will make your tests much slower, and also will probably distort the data generation quite a lot. You should adapt your strategy to filter less. This can also be caused by a low max_leaves parameter in recursive() calls
See https://hypothesis.readthedocs.io/en/latest/healthchecks.html for more information about this. If you want to disable just this health check, add HealthCheck.filter_too_much to the suppress_health_check settings for this test.

While this use of sets/sampled_from is obviously somewhat degenerate, this seems like there may be something strange going wrong (in my actual use case, I'm sampling from a list that just happens to only have one item and may grow to a larger size in the future).

[DRMacIver]

That's... really weird. Literally nothing on the path of that code has changed between 3.17.0 and 3.18.0. Can you double check that it's definitely working on 3.17.0? I can imagine 3.16.1 having broken this, but not 3.18.0.

[alexwlchan]

Also, which system and version of Python are you running this under? I can’t seem to reproduce the error in 2.7 or 3.6 on macOS.

[mithrandi]

Whoops, it turns out that what I thought was a 100% reproducible result is in fact not 100%, and I minimized my test case a tiny bit more reducing the reproducibility a lot. This example hits the problem more often (also on 3.17.0 so I'm not sure any more when this regression hit, probably the big change back in 3.14 or whenever it was):

from hypothesis import assume, given, strategies as s

def flags():
    return s.sets(s.sampled_from([0]))

@given(on=flags(), off=flags())
def test(on, off):
    pass

test()

[DRMacIver]

I'd say with reasonably high confidence that it would be 3.16.1 that introduced this.

[mithrandi]

Seems to be supported experimentally: I ran this in a loop on 3.16.0 without hitting the health check, while on 3.16.1 I hit it fairly often.

[mithrandi]

To answer @alexwlchan's question in case it's still relevant, the updated reproducer worked (although not 100%) on both CPython 2.7.13 and PyPy2 5.8.0.