-rx and -rX should turn on output for xfail/xpass

[okken]

• a detailed description of the bug or problem you are having

• -rx and -rX (as well as -ra and -rA) should allow output for xfail and xpass, but they don't.
• There seems to be no way at all to get the traceback from xfailed tests.
• There seems to be no way to get the output from xfail/xpass. (other than turning all xpass into fail with xpass strict, but that still doesn't solve the xfail case).
• The summary report for xfail doesn't include the assert line, as fail does

This shows up in a discussion #9834 and another issue #10618.

• discussion Why no output sections for Xpass/Xfail? #9834"
  • showing there is confusion in the user base about this
• issue XFail and XPass output does not get printed to console #10618
  • This is tagged "question". It's not a question. It's a defect.
  • @RonnyPfannschmidt notes there that the way around it is to use strict. "That's intentional, With strict, xpass gets treated as error and should have output."
  • That's not a sufficient response because there is still no way to get the output from xfail
  • Even if this is as designed, there need to be a way to see the traceback and output from xfail and xpass. And it seems like -rx and -rX are the obvious choice.

---- More detail about problem and expectations ----

Observations:
Based on a simple test script below that includes test_pass, test_fail, test_xfail, test_xpass.

1. Output and exception for test_fail but not for test_xfail.

   • I would have expected test_xfail to look mostly just like test_fail.
   • If there's a reason to NOT report exception traceback and output for xfails, we should have an option to turn it on.

2. assert 1 == 2 is displayed in summary info for FAILED but not for XFAIL.

   • I can't come up with reason why this would be correct behavior.
   • assert 1 == 2 should show up for XFAIL also.

3. -rP (which is included in -rA) is "pass with output", and it applies to PASSED, but not XPASS.

Opinion on how to fix this:

• XFAIL should act like FAILED if it's turned on with -rx or -ra or -rA
  • output should be reported
  • traceback should be reported
  • assert message should be listed in summary
• XPASS should act like PASSED if it's turned on with -rX or -ra or -rA
  • output should be reported
• It seems reasonable that someone might not want to see all of this extra output.
  • That's why I've suggested that this extra output NOT be on by default.
  • Controlling the extra output with -r flags seems like the right way to do this.

My opinions, of course. But this would follow the idea of "behavior with the least surprise". And it doesn't require any extra flags.

• output of pip list from the virtual environment you are using

$ pip list
Package    Version
---------- -------
colorama   0.4.6
iniconfig  2.0.0
packaging  23.1
pip        23.2
pluggy     1.2.0
pytest     7.4.0
setuptools 65.5.0

• pytest and operating system versions

  • pytest 7.4.0
  • Windows something, but also tested on Mac, so I think OS is irrelevant.

• minimal example if possible

Example: test_foo.py

import pytest

def test_pass():
    print('in test_pass()')
    a, b = 1, 1
    assert a == b

def test_fail():
    print('in test_fail()')
    a, b = 1, 2
    assert a == b

@pytest.mark.xfail
def test_xfail():
    print('in test_xfail()')
    a, b = 1, 2
    assert a == b

@pytest.mark.xfail
def test_xpass():
    print('in test_xpass()')
    a, b = 1, 1
    assert a == b

Current output:

$ pytest -rA test_foo.py
============================= test session starts =============================
platform win32 -- Python 3.11.0, pytest-7.4.0, pluggy-1.0.0
rootdir: C:\Users\okken\projects\instrument_updater
configfile: tox.ini
collected 4 items

test_foo.py .FxX                                                         [100%]

================================== FAILURES ===================================
__________________________________ test_fail __________________________________

    def test_fail():
        print('in test_fail()')
        a, b = 1, 2
>       assert a == b
E       assert 1 == 2

test_foo.py:11: AssertionError
---------------------------- Captured stdout call -----------------------------
in test_fail()
=================================== PASSES ====================================
__________________________________ test_pass __________________________________
---------------------------- Captured stdout call -----------------------------
in test_pass()
=========================== short test summary info ===========================
PASSED test_foo.py::test_pass
XFAIL test_foo.py::test_xfail
XPASS test_foo.py::test_xpass
FAILED test_foo.py::test_fail - assert 1 == 2
============== 1 failed, 1 passed, 1 xfailed, 1 xpassed in 0.28s ==============

[okken]

I'd be happy to work on this, but first I was hoping for

• Agreement that -rx and -rX are the right way to control output for xfail/xpass.
• Some hint at where in the code the changes should go. reports.py?
• A hint at which tests could be used as examples to test this change in behavior

Of course, I'd be "happier" if someone else worked on this. :)

[okken]

Note that this is a significant problem for me currently, and we don't have a workable solution.

[okken]

I'm looking in terminal.py now.
It looks like there are sections for ERRORS, FAILURES, PASSES.
Seems like there needs to be also XFAILURES and XPASSES sections, perhaps after FAILURES and before PASSES.

[okken]

I'm not sure if I'm using labels right. I think this missing functionality is a bug, and I have a proposed way it should work.

[nicoddemus]

Hi @okken,

Sorry for the silence on this, seems like this one fell through the cracks.

Agreement that -rx and -rX are the right way to control output for xfail/xpass.

I think so.

Some hint at where in the code the changes should go. reports.py?

That would go into terminal.py.

A hint at which tests could be used as examples to test this change in behavior

There are several tests in test_terminal.py that handle the -r options.

There's #11574 if you want to comment/help review. 👍