@settings somehow conflates a broken test method with one that doesn't use @given

[Julian]

(In some situations?) @settings thinks that a method hasn't been decorated with given, when instead the issue is simply a broken method signature. See below, where you'll notice I simply forgot to add x and y arguments to the test method itself:

⊙  python -m pip list | rg hypothesis                                                                                                                                julian@Air ●
hypothesis                         4.23.4

~/Desktop
⊙  cat bar.py                                                                                                                                                        julian@Air ●
from unittest import TestCase
from hypothesis import given, settings, strategies


class TestStuff(TestCase):
    @settings(max_examples=1)
    @given(x=strategies.integers(), y=strategies.integers())
    def test_add(self):
        pass

~/Desktop
⊙  trial bar.py                                                                                                                                                      julian@Air ●
bar
  TestStuff
    test_add ...                                                        [ERROR]

===============================================================================
[ERROR]
Traceback (most recent call last):
  File "/usr/local/Cellar/pypy/7.1.0/libexec/lib-python/2.7/unittest/case.py", line 329, in run
    testMethod()
  File "/Users/julian/.local/share/virtualenvs/dev/site-packages/hypothesis/core.py", line 230, in wrapped_test
    raise InvalidArgument(message)
  File "/Users/julian/.local/share/virtualenvs/dev/site-packages/hypothesis/_settings.py", line 255, in new_test
    "Using `@settings` on a test without `@given` is completely pointless."
hypothesis.errors.InvalidArgument: Using `@settings` on a test without `@given` is completely pointless.

bar.TestStuff.test_add
-------------------------------------------------------------------------------
Ran 1 tests in 0.229s

FAILED (errors=1)

After adding them, all's fine:

⊙  trial bar.py                                                                                                                                                      julian@Air ●
bar
  TestStuff
    test_add ...                                                           [OK]

-------------------------------------------------------------------------------
Ran 1 tests in 0.208s

Not sure I see how this is possible given https://github.com/HypothesisWorks/hypothesis/blob/master/hypothesis-python/src/hypothesis/_settings.py#L264 but maybe I'm missing something.

[DRMacIver]

I think I see what's happening. We return a special fake test method when you get the signature of @given wrong, and that's not got is_hypothesis_test set to True on it, so @settings treats it as a non-@given method.

(Side note: I don't like that error message)

[Julian]

(Side note: I don't like that error message)

Agreed :)

[gabrielBusta]

Hi,
If we assign True to is_hypothesis_test in that invalid method hypothesis will throw an exception that is (somewhat) more helpful.

However, adding a flag would be more explicit + it won't affect code reading the is_hypothesis_test flag.

Below is a link to illustrate what I mean.

diff

Below you will find the exception thrown with that change... it is still confusing, it almost seems like the message should read @given got an unexpected keyword argument 'x'.

test_add (exp.TestStuff) ... ERROR

======================================================================
ERROR: test_add (exp.TestStuff)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/gabriel/hypo-dev/hypothesis-python/src/hypothesis/core.py", line 231, in wrapped_test
    raise InvalidArgument(message)
hypothesis.errors.InvalidArgument: test_add() got an unexpected keyword argument 'x'

----------------------------------------------------------------------
Ran 1 test in 0.002s

FAILED (errors=1)

[rsokl]

Below you will find the exception thrown with that change... it is still confusing[...]

Actually, the message:

hypothesis.errors.InvalidArgument: test_add() got an unexpected keyword argument 'x'

is appropriate here. @given is responsible for calling your function/method, so the error message here should be no different as if you had manually called test_add(x=2).

[...] it almost seems like the message should read @given got an unexpected keyword argument x

Reporting that @given got an unexpected keyword argument 'x' would imply that there is something fundamentally wrong with invoking @given(x=st.integers()) under any circumstances.