Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add @example(...).xfail(...) to check inputs which are expected to fail #3530

Closed
Zac-HD opened this issue Dec 11, 2022 · 0 comments · Fixed by #3537
Closed

Add @example(...).xfail(...) to check inputs which are expected to fail #3530

Zac-HD opened this issue Dec 11, 2022 · 0 comments · Fixed by #3537
Assignees
Labels
new-feature entirely novel capabilities or strategies

Comments

@Zac-HD
Copy link
Member

Zac-HD commented Dec 11, 2022

A classic error when testing is to write a test function that can never fail, even on inputs that aren't allowed or manually provided. @rsokl mentioned a nice design pattern for @pytest.mark.parametrize() of including an xfailed param to check that the test can fail; and after some discussion we agreed that this would be nice to have in Hypothesis too.

So: following #3516, I'd like to add an @example(...).xfail(condition: bool = True, *, reason: str = "", raises: type[BaseException] | tuple[...] = BaseException) - matching the interface of pytest.mark.xfail() (omitting the subset that doesn't make sense here). Implementation-wise, this will return self if condition is False, otherwise return an instance of a new XfailExample subclass which is known and handled by the execution logic in core.py. If the exception raised is not an instance of raises and our failure_exceptions_to_catch() it propogates as usual; if nothing is raised then we raise an error (using pytest.xfail() if available). Naturally we'll also support .via().xfail() and .xfail().via() 😁

@Zac-HD Zac-HD added the new-feature entirely novel capabilities or strategies label Dec 11, 2022
@Zac-HD Zac-HD self-assigned this Dec 24, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
new-feature entirely novel capabilities or strategies
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant