Option of treating XPASS as failure? #1355

rabbbit opened this Issue Feb 3, 2016 · 8 comments


None yet

4 participants

rabbbit commented Feb 3, 2016

// enhancement idea
// this might already exist, tried to find it, failed

tldr; I want my xfail'ed tests to cause test suite to fail if they succeed


Often I find myself tests that given an two set outputs, sometimes return a value and sometimes raise specific exceptions. I'm currently splitting this in two separate tests, one for "good" values, one for exceptions, and use pytest.raises.

Inspired by docs on parametrizing tests I thought I could use mark.xfail to group these tests in one logical thing.
However, I don't like XPASS's - I'd like to be able to say that these tests have to fail with specific exceptions.
It would basically be a shorthand for splitting a test into two and using pytest.raises.

Ideally, it would work as an extra argument to xfail (strict=True?), rather than command line option, because I cannot set the value globally.

Basically, I'd like this to fail:

import pytest                                                                                                                                                                     

def complicated_validation_function(input_data):                                                                                                                                  
    if input_data in ('should_raise', 'should_raise_too_but_i_made_a_typo'):                                                                                                      
        raise ValueError                                                                                                                                                          
    return True                                                                                                                                                                   

@pytest.mark.parametrize('input_data', [                                                                                                                                          
def test_validation(input_data):                                                                                                                                                  
    assert complicated_validation_function(input_data) is True 

Does this already exist somewhere?
If not, does this sound like a plausible idea?

edit: I dunno how to mark this as proposal per contributing guidelines


See #1299 (and #814 for lots of background and discussion).

I think there's a general consensus for adding a strict parameter to xfail as you said (and maybe a config option to default to that?), it's just that nobody got to it yet. Want to submit a PR? That'd be very appreciated!

rabbbit commented Feb 3, 2016

Ah, cool, I'm not insane then :)

PR - I'll try to take a look next week, but no promises, I suspect this might be beyond me :)

// what would be really cool would be an option to pass a fixture into .raises=my_fixture, and make it actually compare it but .. lets keep that for later :)

I'll close this one, #1299 is good enough.

@rabbbit rabbbit closed this Feb 3, 2016

the preference for configuration is good, this one it is




It would be nice to introduce this in 2.9... 😁

rabbbit commented Feb 3, 2016

A note that if we only enable this in pytest.ini (which I believe can only be used 1 per project?) I might not be able to use it -> big repo, cannot change default settings without fixing other peoples tests.


@rabbbit thanks for the clarification.

How about this then:

  • pytest.mark.xfail will accept a a strict parameter, which controls if XPASSes fails a test suite. If not given, the value of xfail_strict ini option is used.
  • xfail_strict ini option: default value for the strict parameter of pytest.mark.xfail. In the initial release (say 2.9) this will default to False for backward compatibility, but we will warn users in the release announcement that in the next release (2.10) the default will change to True.

Once this is released, you should add xfail_strict=false to your pytest.ini, so in future pytest releases the current behavior is preserved. You can then use pytest.mark.xfail(strict=True) in particular tests.

Does that cover your needs?

rabbbit commented Feb 3, 2016

Sounds awesome :)

Having an option of setting this per (sub)-directory would also work, but I imagine is out of scope here.

@nicoddemus nicoddemus added a commit to nicoddemus/pytest that referenced this issue Feb 14, 2016
@nicoddemus nicoddemus Add strict option to xfail, making tests which XPASS to actually fail…
… the suite

Fix #1355
@nicoddemus nicoddemus added a commit to nicoddemus/pytest that referenced this issue Feb 15, 2016
@nicoddemus nicoddemus Add CHANGELOG entry for #1355 9aae4b7
@jsonn jsonn pushed a commit to jsonn/pkgsrc that referenced this issue Apr 14, 2016
wiz Update py-test to 2.9.1.

**Bug Fixes**

* Improve error message when a plugin fails to load.
  Thanks `@nicoddemus`_ for the PR.

* Fix (`#1178 <pytest-dev/pytest#1178>`_):
  ``pytest.fail`` with non-ascii characters raises an internal pytest error.
  Thanks `@nicoddemus`_ for the PR.

* Fix (`#469`_): junit parses report.nodeid incorrectly, when params IDs
  contain ``::``. Thanks `@tomviner`_ for the PR (`#1431`_).

* Fix (`#578 <pytest-dev/pytest#578>`_): SyntaxErrors
  containing non-ascii lines at the point of failure generated an internal
  py.test error.
  Thanks `@asottile`_ for the report and `@nicoddemus`_ for the PR.

* Fix (`#1437`_): When passing in a bytestring regex pattern to parameterize
  attempt to decode it as utf-8 ignoring errors.

* Fix (`#649`_): parametrized test nodes cannot be specified to run on the command line.

.. _#1437: pytest-dev/pytest#1437
.. _#469: pytest-dev/pytest#469
.. _#1431: pytest-dev/pytest#1431
.. _#649: pytest-dev/pytest#649

.. _@asottile: https://github.com/asottile


**New Features**

* New ``pytest.mark.skip`` mark, which unconditionally skips marked tests.
  Thanks `@MichaelAquilina`_ for the complete PR (`#1040`_).

* ``--doctest-glob`` may now be passed multiple times in the command-line.
  Thanks `@jab`_ and `@nicoddemus`_ for the PR.

* New ``-rp`` and ``-rP`` reporting options give the summary and full output
  of passing tests, respectively. Thanks to `@codewarrior0`_ for the PR.

* ``pytest.mark.xfail`` now has a ``strict`` option, which makes ``XPASS``
  tests to fail the test suite (defaulting to ``False``). There's also a
  ``xfail_strict`` ini option that can be used to configure it project-wise.
  Thanks `@rabbbit`_ for the request and `@nicoddemus`_ for the PR (`#1355`_).

* ``Parser.addini`` now supports options of type ``bool``.
  Thanks `@nicoddemus`_ for the PR.

* New ``ALLOW_BYTES`` doctest option. This strips ``b`` prefixes from byte strings
  in doctest output (similar to ``ALLOW_UNICODE``).
  Thanks `@jaraco`_ for the request and `@nicoddemus`_ for the PR (`#1287`_).

* Give a hint on ``KeyboardInterrupt`` to use the ``--fulltrace`` option to show the errors.
  Fixes `#1366`_.
  Thanks to `@hpk42`_ for the report and `@RonnyPfannschmidt`_ for the PR.

* Catch ``IndexError`` exceptions when getting exception source location.
  Fixes a pytest internal error for dynamically generated code (fixtures and tests)
  where source lines are fake by intention.


* **Important**: `py.code <http://pylib.readthedocs.org/en/latest/code.html>`_ has been
  merged into the ``pytest`` repository as ``pytest._code``. This decision
  was made because ``py.code`` had very few uses outside ``pytest`` and the
  fact that it was in a different repository made it difficult to fix bugs on
  its code in a timely manner. The team hopes with this to be able to better
  refactor out and improve that code.
  This change shouldn't affect users, but it is useful to let users aware
  if they encounter any strange behavior.

  Keep in mind that the code for ``pytest._code`` is **private** and
  **experimental**, so you definitely should not import it explicitly!

  Please note that the original ``py.code`` is still available in
  `pylib <http://pylib.readthedocs.org>`_.

* ``pytest_enter_pdb`` now optionally receives the pytest config object.
  Thanks `@nicoddemus`_ for the PR.

* Removed code and documentation for Python 2.5 or lower versions,
  including removal of the obsolete ``_pytest.assertion.oldinterpret`` module.
  Thanks `@nicoddemus`_ for the PR (`#1226`_).

* Comparisons now always show up in full when ``CI`` or ``BUILD_NUMBER`` is
  found in the environment, even when ``-vv`` isn't used.
  Thanks `@The-Compiler`_ for the PR.

* ``--lf`` and ``--ff`` now support long names: ``--last-failed`` and
  ``--failed-first`` respectively.
  Thanks `@MichaelAquilina`_ for the PR.

* Added expected exceptions to ``pytest.raises`` fail message.

* Collection only displays progress ("collecting X items") when in a terminal.
  This avoids cluttering the output when using ``--color=yes`` to obtain
  colors in CI integrations systems (`#1397`_).

**Bug Fixes**

* The ``-s`` and ``-c`` options should now work under ``xdist``;
  ``Config.fromdictargs`` now represents its input much more faithfully.
  Thanks to `@bukzor`_ for the complete PR (`#680`_).

* Fix (`#1290`_): support Python 3.5's ``@`` operator in assertion rewriting.
  Thanks `@Shinkenjoe`_ for report with test case and `@tomviner`_ for the PR.

* Fix formatting utf-8 explanation messages (`#1379`_).
  Thanks `@biern`_ for the PR.

* Fix `traceback style docs`_ to describe all of the available options
  (auto/long/short/line/native/no), with `auto` being the default since v2.6.
  Thanks `@hackebrot`_ for the PR.

* Fix (`#1422`_): junit record_xml_property doesn't allow multiple records
  with same name.

.. _`traceback style docs`: https://pytest.org/latest/usage.html#modifying-python-traceback-printing

.. _#1422: pytest-dev/pytest#1422
.. _#1379: pytest-dev/pytest#1379
.. _#1366: pytest-dev/pytest#1366
.. _#1040: pytest-dev/pytest#1040
.. _#680: pytest-dev/pytest#680
.. _#1287: pytest-dev/pytest#1287
.. _#1226: pytest-dev/pytest#1226
.. _#1290: pytest-dev/pytest#1290
.. _#1355: pytest-dev/pytest#1355
.. _#1397: pytest-dev/pytest#1397
.. _@biern: https://github.com/biern
.. _@MichaelAquilina: https://github.com/MichaelAquilina
.. _@bukzor: https://github.com/bukzor
.. _@hpk42: https://github.com/hpk42
.. _@nicoddemus: https://github.com/nicoddemus
.. _@jab: https://github.com/jab
.. _@codewarrior0: https://github.com/codewarrior0
.. _@jaraco: https://github.com/jaraco
.. _@The-Compiler: https://github.com/The-Compiler
.. _@Shinkenjoe: https://github.com/Shinkenjoe
.. _@tomviner: https://github.com/tomviner
.. _@RonnyPfannschmidt: https://github.com/RonnyPfannschmidt
.. _@rabbbit: https://github.com/rabbbit
.. _@hackebrot: https://github.com/hackebrot
This was referenced May 11, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment