New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for pytest xfail concept (variant of skip) #2

Closed
marco-m opened this Issue Aug 1, 2018 · 3 comments

Comments

Projects
None yet
2 participants
@marco-m
Copy link

marco-m commented Aug 1, 2018

Hello,

I really like how this test runner is taking shape, thanks for making it!

From my experience with tests in many languages, two features are important: skip (already supported) and xfail. Quoting from https://docs.pytest.org/en/latest/skipping.html:

A xfail means that you expect a test to fail for some reason. A common example is a test for a feature not yet implemented, or a bug not yet fixed. When a test passes despite being expected to fail (marked with pytest.mark.xfail), it’s an xpass and will be reported in the test summary.

This is very powerful because when a xfail makes the test suite go red, you discover that you might have fixed a bug without knowing it :-) so you can remove the xfail and keep the test as a normal regression.

thanks!

@plexus

This comment has been minimized.

Copy link
Member

plexus commented Aug 1, 2018

This is a cool idea, and the great thing is that this could be implemented as a plugin. Part of the philosophy of Kaocha is that people can experiment with ideas like these, and simply push them to clojars.

Now there's no documentation yet for how to write plugins, and there are probably a few subtleties to look out for here, and this would actually be useful to have in the main distribution. I'll see if I can create a proof of concept.

@plexus

This comment has been minimized.

Copy link
Member

plexus commented Oct 21, 2018

Hi @marco-m, this is now implemented in the :kaocha.plugins.alpha/xfail plugin.

(deftest ^:kaocha/xfail foo-test
  (is false))
bin/kaocha --plugin :kaocha.plugins.alpha/xfail

It currently "inverts" each assertion individually, so a failing (is ,,,) becomes a passing one and vice-versa. I can imagine it becoming smarter in the future by considering the status of the whole test, but this was the easiest to implement.

@plexus plexus closed this Oct 21, 2018

@marco-m

This comment has been minimized.

Copy link

marco-m commented Oct 22, 2018

Thanks a lot @plexus ❤️

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment