filter fails for some lambda functions

[movermeyer]

I stumbled across this bug. For some lambda functions (I've only found this one so far), hypothesis will sometimes fail.

This is the best test case I've been able to develop thus far:

from hypothesis import given
from hypothesis.strategies import text

@given(text(min_size=1).filter(lambda x: not x.endswith('.')))
def go(value1):
    pass

go()

Then env/bin/python test.py, with some probability (about 20% on my machine), causes the following exception:

Traceback (most recent call last):
  File "test.py", line 8, in <module>
    go()
  File "test.py", line 5, in go
    def go(value1):
  File "<redacted>/env/lib/python2.7/site-packages/hypothesis/core.py", line 476, in wrapped_test
    runner.run()
  File "<redacted>/env/lib/python2.7/site-packages/hypothesis/internal/conjecture/engine.py", line 191, in run
    self._run()
  File "<redacted>/env/lib/python2.7/site-packages/hypothesis/internal/conjecture/engine.py", line 339, in _run
    self.test_function(data)
  File "<redacted>/env/lib/python2.7/site-packages/hypothesis/internal/conjecture/engine.py", line 92, in test_function
    self.note_details(data)
  File "<redacted>/env/lib/python2.7/site-packages/hypothesis/internal/conjecture/engine.py", line 146, in note_details
    for event in set(map(self.event_to_string, data.events)):
  File "<redacted>/env/lib/python2.7/site-packages/hypothesis/internal/conjecture/engine.py", line 495, in event_to_string
    result = str(event)
  File "<redacted>/env/lib/python2.7/site-packages/hypothesis/internal/deferredformat.py", line 29, in __str__
    return self.__format_string % self.__args
  File "<redacted>/env/lib/python2.7/site-packages/hypothesis/searchstrategy/strategies.py", line 287, in __repr__
    self.condition)
  File "<redacted>/env/lib/python2.7/site-packages/hypothesis/internal/reflection.py", line 228, in get_pretty_function_description
    result = extract_lambda_source(f)
  File "<redacted>/env/lib/python2.7/site-packages/hypothesis/internal/reflection.py", line 181, in extract_lambda_source
    result = _extract_lambda_source(f)
  File "<redacted>/env/lib/python2.7/site-packages/hypothesis/internal/reflection.py", line 199, in _extract_lambda_source
    out=out, is_pypy=PYPY, compile_mode='eval'
  File "<redacted>/env/lib/python2.7/site-packages/uncompyle6/semantics/pysource.py", line 2450, in deparse_code
    tokens, customize = scanner.ingest(co, code_objects=code_objects)
  File "<redacted>/env/lib/python2.7/site-packages/uncompyle6/scanners/scanner2.py", line 80, in ingest
    n = self.setup_code(co)
  File "<redacted>/env/lib/python2.7/site-packages/uncompyle6/scanners/scanner2.py", line 303, in setup_code
    for i in self.op_range(0, len(self.code)):
  File "<redacted>/env/lib/python2.7/site-packages/uncompyle6/scanner.py", line 220, in op_range
    start += self.op_size(self.code[start])
  File "<redacted>/env/lib/python2.7/site-packages/uncompyle6/scanners/scanner2.py", line 289, in op_size
    if op < self.opc.HAVE_ARGUMENT and op not in self.opc.hasArgumentExtended:
AttributeError: 'module' object has no attribute 'hasArgumentExtended'

I'm running hypothesis (3.5.3) on CentOS 7, with Python 2.7.5 in a virtualenv.

Aside: This happens to occur in proximity to this possibly telling comment.

Another Aside: Github doesn't list 3.5.3 in the releases tab, but it exists on PyPi

[DRMacIver]

Hmm. That's unfortunate. It's probably an upstream bug in uncompyle6, but it might be an oddity in how I'm invoking it.

I'm on a break at the moment, so I'm afraid I'm probably not going to be going to looking into this in depth in the near future. In the meantime I'd recommend working around it by defining a named function. Sorry.

PS. I've fixed the release tag issue. Thanks for pointing it out.

[DRMacIver]

Oh, BTW, I think the reliable way to trigger this is going to be to call repr() on the strategy object. The random factor will be because that only happens in a small fraction of test runs because that filter is usually satisfied.

[movermeyer]

Ah, OK. That explains the non-determinism.

Here is a reduced test case that works every time:

from hypothesis.strategies import just
while True:
    just('.').filter(lambda x: not x.endswith('.')).example()

Also, the suggested workaround of named function does not work:

from hypothesis.strategies import just
suggested_workaround_i_think = lambda x: not x.endswith('.')
while True:
    just('.').filter(suggested_workaround_i_think).example()

still fails in the same way.

[movermeyer]

PS: I understand and sympathize with the reasoning for your break. Please don't feel any obligations. I'm just posting this bug so that it is known.

[DRMacIver]

Ah, no, the suggested workaround is:

def workaround(x):
    return not x.endswith('.')

@given(text(min_size=1).filter(workaround))
def go(value1):
    pass

The bug comes from when Hypothesis tries to get a string representation for the lambda, which it won't do if the function object has a sensible name.

And please do feel free to continue to post bugs, and thanks for reporting them! I'll probably still respond to them and will definitely look at them properly when I'm back on duty :-)

[ashb]

@movermeyer I think you need to make it not-a-lambda like this:

from hypothesis.strategies import just

def suggested_workaround_i_think(x):
     return not x.endswith('.')

while True:
    just('.').filter(suggested_workaround_i_think).example() 

(Excuse the formatting, doing this on a phone)

[movermeyer]

Ah, thanks. Much better.

[ashb]

Beaten to the punch :)

[Julian]

I started seeing this error consistently, and then suddenly it's disappeared for me as well :/. I'll try to follow up with my own example as soon as I can get it to happen again.

[DRMacIver]

So I've just observed that the Hypothesis build is also seeing this issue consistently. This pretty strongly suggests that it's the result of a version bump in uncompyle6.

If someone wants to debug this and post a bug upstream that would be lovely. :-) If not, pinning to an older version of uncompyle6 is probably also a viable workaround.

This might also be a sign that Hypothesis adding a dependency on uncompyle6 was a bad move on my part. It was always a bit of an unsupported use case for the tool.

[alexwlchan]

I just tried to repro this issue on macOS, running Python 3.5.0 and a variety of uncompyle6 versions, and I’m not seeing any issues. 😕

But there was an uncompyle6 version bump right around the time this started happening (timestamp on @movermeyer’s original post is 25 October, PyPI says 26 October, but timezone interactions are weird).

I’ve opened a PR that pins the version to get running against Hypothesis CI. Maybe that will have better luck finding (or not finding?) the problem.

[alexwlchan]

@movermeyer Could I get the output of pip freeze inside your virtualenv please?

[movermeyer]

@alexwlchan:

> env/bin/pip freeze

Cython==0.24.1
enum34==1.1.6
funcsigs==1.0.2
hypothesis==3.5.3
mock==2.0.0
pbr==1.10.0
py==1.4.31
pyhocon==0.3.33
pyparsing==2.1.9
pytest==3.0.3
python-afl==0.5.6
six==1.10.0
spark-parser==1.4.0
uncompyle6==2.9.2
wxPython==3.0.2.0
xdis==3.2.0

[alexwlchan]

At last, a successful Travis run in #381! So it seems like the issue is somewhere xdis 3.2.0, and we need the following pins:

pip install xdis==3.1.0 uncompyle6==2.9.2

@movermeyer Could you try installing those versions and let me know if that resolves the issue for you?

---

Assuming my pins are correct, we need to:

• Add the test cases from this issue to CI, so we can spot it if it regresses again.
• Update setup.py. I think we should pin the exact versions in install_requires. And make sure we document the interaction – in the unlikely event somebody needs a different version of uncompyle6 elsewhere in their test environment, we should have an easy-to-find explanation of the pin.
• Blow away the Travis cache – that already contains xdis 3.2.0, and tox is antsy about the conflicting versions.
• Cut a 3.5.4 release that pins the version. There haven’t been any code changes on master since we cut 3.5.3, so that’s a pretty safe change.
• File an upstream bug report with xdis about the bug. Skimming their commit logs, it seems like they did quite a bit of Python 2.7 work in the latest release. I might dig into it a bit further, or I might just chuck this bug over the fence and let them debug it themselves. Let’s see how generous I’m feeling.

Does that sound like a sensible plan @DRMacIver?

[alexwlchan]

• Investigate pinning the versions in the oldpy27 tox environment.

We install uncompyle6 as an explicit dependency in the oldpy27 tox environment. I’m not entirely sure why the test passed without my extra pin, but given what uncompyle6 is up to, I could believe this is the sort of subtle bug that might affect different minor versions of Python.

https://github.com/HypothesisWorks/hypothesis-python/blob/master/tox.ini#L34

We should either add another pin to be safe, or remove this dependency in tox.ini. I think it should be installed automatically by setup.py, and if it turns out not to be the case, I want to add a comment explaining why.

[alexwlchan]

I’m now very confident this is an upstream bug that’s fixed by pinning dependencies. (Something I could have figured out much quicker if I’d actually read the exception message!)

Hopefully just a small change upstream, and then Hypothesis will be back to rights. Hooray!

[alexwlchan]

Good news and bad news:

• The good news is that the hasArgumentExtended bug seems to be entirely fixed by upgrading to uncompyle6 v2.9.3. Ironically, this was released just as this issue was raised, which may explain why I was unable to reproduce it locally (that and caching). As that update filters out, this bug will disappear from the wild without any action on our part. Hooray!
• Bad news: if you look at the Travis CI run that has uncompyle6 v2.9.3, it’s failing when it gets the repr() of some flatmap strategies, albeit only on PyPy. I’ve traced the problem to a single commit in xdis 3.2.0, which I’ll raise as a separate issue and put together a report for upstream.

I think there remains a strong argument for restricting the max version of dependencies until they’ve been qualified in Hypothesis CI. Less hassle than an explicit version pin, but makes it easier to spot this sort of bug

[alexwlchan]

With apologies for a slightly roundabout debugging approach, I believe this is now fixed.

We have a clean CI run with uncompyle 2.9.3 and xdis 3.2.1, which does not feature this hasArgumentExtended error.

@movermeyer Please can you run pip install --upgrade uncompyle6 xdis in your virtualenv, and see if that fixes the issue?

[movermeyer]

@alexwlchan, that seems to work.

Thanks for putting in the time to investigate this.

[DRMacIver]

...but I'm about to invalidate most of @alexwlchan's hard work (sorry again Alex, but thanks for doing it anyway as I wouldn't have noticed this without it). :-(

It turns out that the uncompyle6 dependency snuck in some GPL code under the radar. Between that (and my being already predisposed to do so by this issue), I've released Hypothesis 3.6.0 which restores the code that uncompyle6 was used to replace and drops the uncompyle6 dependency. You should probably upgrade to that.