Conditional skip and xfail in doctests

[pytestbot]

Originally reported by: Danilo Bellini (BitBucket: danilobellini, GitHub: danilobellini)

---

I'd like to skip or xfail some doctests, since they depend on something that doesn't exist in some possible system configuration. However, as I tried something like:

:::python
    def anything():
        """
        >>> import pytest
        >>> pytest.xfail()
        >>> raise Exception("Just anything...")
        """

It happens to fail.

============================= test session starts ==============================
platform linux2 -- Python 2.7.3 -- pytest-2.3.5
plugins: timeout, xdist, cov
collected 1 items

pytest_skip_xfail_docstring.py F

=================================== FAILURES ===================================
__________________________________ [doctest] ___________________________________
009     """
010     >>> import pytest
011     >>> pytest.xfail()
UNEXPECTED EXCEPTION: <XFailed instance>
Traceback (most recent call last):

  File "/usr/lib/python2.7/doctest.py", line 1289, in __run
    compileflags, 1) in test.globs

  File "<doctest pytest_skip_xfail_docstring.anything[1]>", line 1, in <module>

  File "/usr/local/lib/python2.7/dist-packages/_pytest/skipping.py", line 37, in xfail
    raise XFailed(reason)

XFailed: <XFailed instance>

/media/danilo/CODE/Spyder Workspace/pytest_skip_xfail_docstring.py:11: UnexpectedException
=========================== 1 failed in 0.33 seconds ===========================

I can use # doctest: +SKIP in doctests, and make a manually-written change into each __doc__ (besides class docstrings) to at least skip problematic tests, but that's far from a nice solution (and there couldn't be any xfail at all).

I've tried to change DoctestModule in _pytest/doctest.py by decorating it (monkeypatching) to call pytest.xfail() when exc_info[0] is a XFailed instance, however:

=================================== FAILURES ====================================
___________________________________ [doctest] ___________________________________
test/__init__.py:119: in new_meth
>         pytest.xfail(e.exc_info[1])
E         XFailed: This reason is just a string

It seems doctests can't xfail nor skip (besides doctest directive). This proposal is to allow [conditional] skips and xfails in doctests.

---

• Bitbucket: https://bitbucket.org/pytest-dev/pytest/issue/310

[RonnyPfannschmidt]

Closing as stalled

[jaraco]

I'd like to see this feature also. The Sphinx test runner added such support here. I need this feature to help address jaraco/jaraco.functools#12, where a small bug in the Python interpreter exists for only one patch Python version (3.7.3). It would be nice to be able to skip that test on Python 3.7.3.

[pjadzinsky]

I had a similar issure recently and solve it like this. Can't you just do the same? You just redefine the decorator to do nothing in certain cases (where the test should run ...

decorator = pytest.mark.xfail
if <condition where test passes>
    def decorator(f):
        return f

in your case it would be more like

if python_matches_3.7.3:
   cond_xfail = pytest.mark.xfail
else:
   def cond_xfail(f):
      return f

And decorate all the tests with cond_xfail

[jaraco]

@pjadzinsky This issue is about doctests, where tests are organized by command/output assertions, so decorator syntax isn't relevant.

[blueyed]

pytest.skip() should work already.
#6669 adds support for all outcome exceptions.

btw: there's also a SKIP directive/option (https://docs.python.org/3/library/doctest.html?highlight=doctest#option-flags / https://docs.python.org/3/library/doctest.html?highlight=doctest#directives), but it is not dynamic.
Having import pytest … in the docs is not that nice after all, is it? :)

[jaraco]

What I'm hoping for is something where a conditional expression can be added such that a test that needs to be skipped or xfailed can be tagged as such, something like:

def somefunc():
    """
    >>> sys.version_info  # doctest.SKIP_IF sys.version_info != (3, 7, 3)
    (3, 7, 3)
    >>> isinstance(sys.version_info, tuple)
    True
    """

So in the same way that skip would unconditionally skip just that line from the doctest, this test would by skipped if the supplied expression evaluated to false. I would expect the rest of the doctest to continue to run (so isinstance check would be run unconditionally).

[blueyed]

@jaraco
Makes sense. But that seems to be more like a doctests feature in general (i.e. for Python itself).
Have you checked their issue tracker?
(I've not looked much into the doctests plugin yet, but we could probably extend it ourselves though, of course)

so isinstance check would be run unconditionally

IIRC SKIP would skip the whole block, so this seems a bit inconsistent then. SKIP_LINE_IF could be used though I guess.