@xfail annotation

[MikkelPaulson]

A useful feature of pytest is its @pytest.mark.xfail annotation, which indicates that a test should run but is expected to fail. If a test so annotated does pass, the test run interprets it as a failure.

Generally this is used in TDD and BDD to indicate that a bug or feature is still a work in progress. It's distinct from skipping in that the failure when the test starts passing should prompt the developer to remove the annotation or update the test accordingly.

I did some looking around and was surprised to see that this seems to be a unique feature of pytest (at least by that name). I'd like to add it to PHPUnit, but wanted to get a 👍 👎 before starting work.

[sebastianbergmann]

I am open to the idea but I do think we should deprecate (and later remove) markTestIncomplete().

[MikkelPaulson]

I'm not sure that they're exactly synonymous. @xfail indicates that the feature is incomplete, and markTestIncomplete() indicates that test coverage is missing/deficient and needs to be updated – at least in theory. They're both ways of saying that the test and the code are out of sync, but they do have different meanings and behaviour. I feel like in most real-world projects incomplete tests are a sloppy way of saying "I don't feel like testing this" and the annotation will remain until the test is discarded completely, but maybe I should have more faith in people. I'm not opposed to deprecating it, though. It doesn't do anything you couldn't do by ending your test with $this->skip("incomplete").

[morozov]

FWIW, besides being often abused, markTestIncomplete() is useful in the cases where the same test is run against multiple implementations and some of them are known to fail. This way, an incomplete test documents a known issue.

[remorhaz]

This feature can be useful for testing of different PHPUnit constraints and extensions. It would be very convenient just to annotate that a concrete test "should fail".

[sebastianbergmann]

Won't implement.

[weitzman]

Any chance yu are open to this in some way other than an @xfail annotation? markTestIncomplete() isnt a great substitute IMO.