Full diff thinks unittest.mock.ANY fails equality check in assertions

[Fak3]

When unittest.mock.ANY is used, and assertion fails, the pytest diff is confusing, and it mistakenly marks ANY being non-equal to the value on the other side. This is especially confusing with lists comparison. Simple example:

from unittest.mock import ANY
assert [{'id': 1, 'val': 5}] == [{'id': ANY, 'val': 4}]

The full diff produced with pytest -vv:

E       AssertionError: assert [{'id': 1, 'val': 5}] == [{'id': <ANY>, 'val': 4}]
E         At index 0 diff: {'id': 1, 'val': 5} != {'id': <ANY>, 'val': 4}
E         Full diff:
E         - [{'id': 1, 'val': 5}]
E         ?         ^         ^
E         + [{'id': <ANY>, 'val': 4}]
E         ?         ^^^^^         ^

In the full diff you can see that not only val, but also id is marked as being different, which is confusing, as it does not trigger the assertion. It would be a lot more clear if full diff only show the values actually comparing non-equal

pytest version: 3.6.2

[pytestbot]

GitMate.io thinks possibly related issues are #381 (Option to show full assertion diff), #715 (Full diff of sequences should be displayed on assertion failures when verbose), #2769 (from unittest.mock import call), #2040 (isfile check fails within pytest), and #79 (assertion rewriting fails with 'in' and 'and' operators).

[asottile]

I've always found this a little confusing as well but it's also nearly impossible to fix. The "Full diff" that pytest prints here is essentially just a diff of the repr(...) of the two objects. pytest doesn't know any better about the actual contents of the objects in play here.

I agree that this is non-ideal, but I also don't think this is solvable.

[blueyed]

Yes, it is a well-known issue, and can be confusing with large dicts especially.

The differ could maybe be a bit smarter, and inspect each key and value in the dict by itself recursively?

[asottile]

It does do that for dictionaries, though for OP's case the differ is looking at a list of objects.

____________________________________ test2 _____________________________________

    def test2():
>       assert {'a': mock.ANY, 'b': 1} == {'a': '2', 'b': 2}
E       AssertionError: assert {'a': <ANY>, 'b': 1} == {'a': '2', 'b': 2}
E         Omitting 1 identical items, use -vv to show
E         Differing items:
E         {'b': 1} != {'b': 2}
E         Use -v to get the full diff

t.py:20: AssertionError

[blueyed]

Yeah, it should do this recursively for iterables somehow.
E.g. after the "At index 0 diff".

The "Full diff" could still be done like it's done for now in the end, but having the explicit "Differing items" might help to avoid confusion (and makes it easier with large diffs).