Misleading output with verbose "-v" caused by repr.

[WloHu]

• a detailed description of the bug or suggestion

Code example shows test file and output for which pytest in misleading way shows the cause of error when verbose (pytest -v) is used. The reason is that equal items have different repr but faulty ones text representation is the same.

The output is misleading because the real cause of assertion failure is another item (item with None in example) not the item which ++ indicates.

This is especially confusing when (in this case) outermost list contains more than 2 items (just duplicate lines with Est(None) like 10+ times) because the faulty/other items are merged and equal ones with different repr are displayed like they are the cause of failure.

Output with less verbosity (w/o -v) is more accurate. More verbose levels than -v are equally confusing.

• output of pip list from the virtual environment you are using

asn1crypto         0.24.0
atomicwrites       1.3.0
attrs              19.1.0
beautifulsoup4     4.7.1
certifi            2019.6.16
cffi               1.12.3
chardet            3.0.4
Click              7.0
coverage           4.5.3
cryptography       2.7
decorator          4.4.0
defusedxml         0.6.0
idna               2.8
importlib-metadata 0.18
Jinja2             2.10.1
jira               2.0.0
MarkupSafe         1.1.1
more-itertools     7.1.0
networkx           2.2
oauthlib           3.0.2
packaging          19.0
pbr                5.4.0
pip                19.1.1
pluggy             0.12.0
py                 1.8.0
pycparser          2.19
PyJWT              1.7.1
pyparsing          2.4.0
pytest             5.0.1
pytest-cov         2.7.1
PyYAML             5.1
requests           2.22.0
requests-oauthlib  1.2.0
requests-toolbelt  0.9.1
setuptools         41.0.1
six                1.12.0
soupsieve          1.9.2
urllib3            1.25.3
wcwidth            0.1.7
wheel              0.33.4
zipp               0.5.2

• pytest and operating system versions
  OS 5.1.14-arch1-1-ARCH
  pytest 4.1.0; 5.0.1

• minimal example if possible

Notice comments in code.

#test_.py
from contextlib import suppress
from typing import Optional, Union
import pytest

class Est:
    def __init__(self, value: Optional[Union[float,int]]):
        self.value = value
    def __repr__(self):
        return f'{type(self).__name__}(value={self.value})'
    def __eq__(self, other):
        if isinstance(other, type(self)):
            other = other.value
        # FIXME sic! Missing case for comparing `None`, thus `Est(None) == Est(None)` -> `False`.
        with suppress(Exception):
            return other == pytest.approx(self.value)
        return False

def test_():
    given = [
        ('a', Est(1)),
        ('b', Est(None)),
    ]
    expected = [
        ('a', Est(1.)),  # `1 == 1.` is NOT the real failure reason as "-vvv" shows.
        ('b', Est(None)),
    ]
    assert given == expected

The most accurate output w/o -v.

$ .venv/python/p3/bin/python -m pytest sandbox/pytest_wrong_fail_output/
========================================== test session starts ===========================================
platform linux -- Python 3.6.6, pytest-5.0.1, py-1.8.0, pluggy-0.12.0
rootdir: /home/user
plugins: cov-2.7.1
collected 1 item

sandbox/pytest_wrong_fail_output/test_.py F                                                        [100%]

================================================ FAILURES ================================================
_________________________________________________ test_ __________________________________________________

    def test_():
        given = [
            ('a', Est(1)),
            ('b', Est(None)),
            ('b', Est(None)),
            ('b', Est(None)),
        ]
        expected = [
            ('a', Est(1.)),  # `1 == 1.` is NOT the real failure reason as "-vvv" shows.
            ('b', Est(None)),
            ('b', Est(None)),
            ('b', Est(None)),
        ]

>       assert given == expected
E       AssertionError: assert [('a', Est(va...(value=None))] == [('a', Est(val...(value=None))]
E         At index 1 diff: ('b', Est(value=None)) != ('b', Est(value=None))
E         Use -v to get the full diff

/home/user/sandbox/pytest_wrong_fail_output/test_.py:34: AssertionError
======================================== 1 failed in 0.06 seconds ========================================

Confusing output when -vvv is used with less items.

$ /home/user/.venv/python/p3/bin/python -m pytest sandbox/pytest_wrong_fail_output/ -vvv  # MAX VERBOSE!
========================================== test session starts ===========================================
platform linux -- Python 3.6.6, pytest-5.0.1, py-1.8.0, pluggy-0.12.0 -- /home/user/.venv/python/p3/bin/python
cachedir: .pytest_cache
rootdir: /home/user
plugins: cov-2.7.1
collected 1 item

sandbox/pytest_wrong_fail_output/test_.py::test_ FAILED                                            [100%]

================================================ FAILURES ================================================
_________________________________________________ test_ __________________________________________________

    def test_():
        given = [
            ('a', Est(1)),
            ('b', Est(None)),
        ]
        expected = [
            ('a', Est(1.)),
            ('b', Est(None)),
        ]  # `1 == 1.` is NOT the real failure reason as "-vvv" shows.

>       assert given == expected
E       AssertionError: assert [('a', Est(va...(value=None))] == [('a', Est(val...(value=None))]
E         At index 1 diff: ('b', Est(value=None)) != ('b', Est(value=None))
E         Full diff:
E         - [('a', Est(value=1)), ('b', Est(value=None))]
E         + [('a', Est(value=1.0)), ('b', Est(value=None))]
E         ?                   ++

/home/user/sandbox/pytest_wrong_fail_output/test_.py:30: AssertionError
======================================== 1 failed in 0.06 seconds ========================================

Even more confusing output when there is more items with matching repr.

$ .venv/python/p3/bin/python -m pytest ~/sandbox/pytest_wrong_fail_output/ -v
========================================== test session starts ===========================================
platform linux -- Python 3.6.6, pytest-5.0.1, py-1.8.0, pluggy-0.12.0 -- /home/user/.venv/python/p3/bin/python
cachedir: .pytest_cache
rootdir: /home/user
plugins: cov-2.7.1
collected 1 item

sandbox/pytest_wrong_fail_output/test_.py::test_ FAILED                                            [100%]

================================================ FAILURES ================================================
_________________________________________________ test_ __________________________________________________

    def test_():
        given = [
            ('a', Est(1)),
            ('b', Est(None)),
            ('b', Est(None)),
            ('b', Est(None)),
        ]
        expected = [
            ('a', Est(1.)),  # `1 == 1.` is NOT the real failure reason as "-vvv" shows.
            ('b', Est(None)),
            ('b', Est(None)),
            ('b', Est(None)),
        ]

>       assert given == expected
E       AssertionError: assert [('a', Est(va...(value=None))] == [('a', Est(val...(value=None))]
E         At index 1 diff: ('b', Est(value=None)) != ('b', Est(value=None))
E         Full diff:
E         - [('a', Est(value=1)),
E         + [('a', Est(value=1.0)),
E         ?                   ++
E         ('b', Est(value=None)),
E         ('b', Est(value=None)),...
E
E         ...Full output truncated (2 lines hidden), use '-vv' to show

/home/user/sandbox/pytest_wrong_fail_output/test_.py:34: AssertionError
======================================== 1 failed in 0.06 seconds ========================================

[blueyed]

Thanks for the report - have not read it fully, but wanted to suggest trying it with --assert=plain also.

[WloHu]

Generic assertion message from --assert=plain is even less helpful.

>       assert given == expected
E       AssertionError

[asottile]

there's several duplicates for this somewhere that are difficult to find #3638 is one such duplicate -- basically pytest is just diffing the repr(...)s of the two objects and has no insight into their contents other than the top level objects are unequal

it's unlikely to be solvable given pytest would need to know about the implementation of the underlying objects being compared

the most common case I've run into for this is comparing against large object structures where one of the fields is mock.ANY (this will always lead to pytest's diff marking that as different even though other parts of the object are the actual different parts)

[WloHu]

Thanks for linking. I couldn't find it because I was looking for "repr" in issues titles.