Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Verbosity setting not always respected #944

Closed
Zac-HD opened this issue Oct 16, 2017 · 5 comments · Fixed by #1580
Closed

Verbosity setting not always respected #944

Zac-HD opened this issue Oct 16, 2017 · 5 comments · Fixed by #1580
Assignees
Labels
bug something is clearly wrong here

Comments

@Zac-HD
Copy link
Member

Zac-HD commented Oct 16, 2017

This is a fairly subtle bug I found while working on #931.

After fixing the Django tests to use Dajngo-2.0 idioms, I ran into a weird issue with passing tests printing an @seed note even when they passed.

$python -m tests.django.manage test tests.django.toystore.test_basic_configuration
OK
Creating test database for alias 'default'...
System check identified no issues (0 silenced).
You can add @seed(301236684161683320861498655448715075617) to this test to reproduce this failure.
Destroying test database for alias 'default'...

This happens because the inner test throws an error in example generation, therefore fails and prints a note, then passes because that's what the outer test wanted to see. The problem is that adding @setting(verbosity=Verbosity.quiet) on the inner test should disable this message, but currently doesn't.

My initial fix is here - note that I had to use a with settings(...) because simply adding it to the test settings didn't work.

@Zac-HD Zac-HD added the bug something is clearly wrong here label Oct 16, 2017
@Zac-HD Zac-HD changed the title Odd Verbosity setting not always respected Oct 16, 2017
@Zac-HD
Copy link
Member Author

Zac-HD commented Nov 1, 2017

I think the correct fix is to check the verbosity setting here:

https://github.com/HypothesisWorks/hypothesis-python/blob/ba9dd88b3bb339d2d1d1135a6f20e33b7d51185c/src/hypothesis/core.py#L1023-L1025

Nope, report should already be doing that. Hmm... My next guess is it's something about defining the test inside our test function, or we'd see this much more often.

@Zac-HD
Copy link
Member Author

Zac-HD commented Nov 1, 2017

Hmm. This doesn't happen under Python 2.7, at least on Travis. Suggests that investigating version differences is likely to be important.

@rbdixon
Copy link

rbdixon commented Mar 5, 2018

I'm not certain if this is related but under Python 3.6.1 I have pytest test functions that will dynamically skip certain tests. When a test is skipped the seed message is displayed even when verbosity is set to quiet. This isn't the behavior I expected. If I can help with additional details please let me know.

@dchudz
Copy link
Member

dchudz commented Mar 7, 2018

Hmm, @rbdixon, an example of that seems like it might be pretty useful if you can provide one.

@Zac-HD
Copy link
Member Author

Zac-HD commented Sep 15, 2018

Found the bug; the relevant part of core.py (in wrapped_test) is passing around an explicit settings object to everything except those two calls to report(). I'll open a PR soon.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug something is clearly wrong here
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants