Diff for visually comparing actual with expected in mock.assert_called_with.

[EliRose]

BPO	28054
Nosy	@rhettinger, @cjw296, @voidspace, @asfaltboy, @mariocj89, @tirkarthi
Files	issue28054.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2016-09-09.22:44:33.486>
labels = ['3.8', 'type-feature', 'tests']
title = 'Diff for visually comparing actual with expected in mock.assert_called_with.'
updated_at = <Date 2018-12-11.06:33:15.547>
user = 'https://bugs.python.org/EliRose'

bugs.python.org fields:

activity = <Date 2018-12-11.06:33:15.547>
actor = 'cjw296'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Tests']
creation = <Date 2016-09-09.22:44:33.486>
creator = 'Eli Rose'
dependencies = []
files = ['47982']
hgrepos = []
issue_num = 28054
keywords = ['patch']
message_count = 7.0
messages = ['275481', '279319', '279394', '329636', '329653', '331433', '331583']
nosy_count = 7.0
nosy_names = ['rhettinger', 'cjw296', 'michael.foord', 'asfaltboy', 'Eli Rose', 'mariocj89', 'xtreak']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue28054'
versions = ['Python 3.8']

[EliRose]

When I call unittest.TestCase.assertEqual(a, b) on e.g. two unequal dictionaries, I get a nice diff pointing me to the differences.

>>> class A(unittest.TestCase):
...     def test_foo(self):
...         self.assertEqual(dict(foo='bar', zab='zar'), dict(foo='bar', zab='zab'))
>>> unittest.main()

======================================================================
FAIL: test_foo (main.A)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "<stdin>", line 3, in test_foo
AssertionError: {'foo': 'bar', 'zab': 'zar'} != {'foo': 'bar', 'zab': 'zab'}
- {'foo': 'bar', 'zab': 'zar'}
?                          ^

+ {'foo': 'bar', 'zab': 'zab'}
? ^

----------------------------------------------------------------------
Ran 1 test in 0.001s

FAILED (failures=1)

But when unittest.mock.Mock.assert_called_with fails, I don't get this nice diff output.

This would be very helpful in my present case (asserting that a function with many keyword arguments is called correctly).

[asfaltboy]

An implementation of this exists in pytest-mock. Recently an idea was brought up to shift the development of PR #57 into mock directly for everyone's benefit.

https://github.com/pytest-dev/pytest-mock
pytest-dev/pytest-mock#58 (comment)

[voidspace]

I like the idea and would be happy for it to be added to mock.

[tirkarthi]

I think this can be useful for keyword arguments as in the original suggestion. I tried an initial implementation as below to use difflib to get the difference like unittest if there are keyword args to be checked against the caller list since the absence of one or other generates a diff with empty {} which I find little distracting during my initial iterations. I would like to know if there is still sufficient interest in getting this to core only for keyword arguments given that there is pytest-mock with specialized error reporting handling more cases : https://github.com/pytest-dev/pytest-mock#improved-reporting-of-mock-call-assertion-errors

$ ./python.exe -q
>>> from unittest.mock import Mock
>>> m = Mock()
>>>
>>> m(foo='bar', bar='baz')
<Mock name='mock()' id='4484887184'>
>>>
>>> m.assert_called_with(bar='baz', foo='car')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/unittest/mock.py", line 838, in assert_called_with
    raise AssertionError(_error_message()) from cause
AssertionError: Expected call: mock(bar='baz', foo='car')
Actual call: mock(bar='baz', foo='bar')
- {'bar': 'baz', 'foo': 'car'}
?                        ^

+ {'bar': 'baz', 'foo': 'bar'}
? ^

diff --git a/Lib/unittest/mock.py b/Lib/unittest/mock.py
index a9c82dcb5d..8603b4ac4c 100644
--- a/Lib/unittest/mock.py
+++ b/Lib/unittest/mock.py
@@ -749,6 +749,20 @@ class NonCallableMock(Base):
         call_args = self.call_args
         if len(call_args) == 3:
             call_args = call_args[1:]
+
+        diffMsg = ''
+        if kwargs and self.call_args[1]:
+            import difflib
+
+            seq1 = kwargs
+            seq2 = self.call_args[1]
+            diffMsg = '\n' + '\n'.join(
+                difflib.ndiff(pprint.pformat(seq1).splitlines(),
+                              pprint.pformat(seq2).splitlines()))
+
+        if diffMsg:
+            message += diffMsg
+
         actual_string = self._format_mock_call_signature(*call_args)
         return message % (expected_string, actual_string)

[rhettinger]

+1 for extending the unittest display niceties to mock.

[tirkarthi]

It seems that TestCase in unittest.case accepts failureException attribute that is raised on assertion failure. My initial approach is to subclass TestCase and add a custom failure exception like MockException that is caught to add the diff to the original message. I am adding @cjw296 and @mariocj89 for feedback since I hope this can reuse unittest.case.TestCase implementation for diffs. I think this is better done with a flag like verbose=True to display the difference and defaulting to False since sometimes the message can be long depending on args and kwargs. Also I find args (tuple) comparison to be little distracting. Since this is an enhancement I am adding 3.8.

# Sample file

from unittest.mock import Mock

m = Mock()
m(1, 2, foo='bar', bar='baz')
m.assert_called_with(2, 3, bar='baz', foo='car')

# On 3.7

$ python3.7 /tmp/foo_4.py
Traceback (most recent call last):
  File "/tmp/foo_4.py", line 5, in <module>
    m.assert_called_with(2, 3, bar='baz', foo='car')
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/unittest/mock.py", line 820, in assert_called_with
    raise AssertionError(_error_message()) from cause
AssertionError: Expected call: mock(2, 3, bar='baz', foo='car')

# With patch

$ ./python.exe /tmp/foo_4.py
Traceback (most recent call last):
  File "/tmp/foo_4.py", line 5, in <module>
    m.assert_called_with(2, 3, bar='baz', foo='car')
  File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/unittest/mock.py", line 844, in assert_called_with
    raise AssertionError(_error_message()) from cause
AssertionError: Expected call: mock(2, 3, bar='baz', foo='car')
Actual call: mock(1, 2, bar='baz', foo='bar')
args: Tuples differ: (2, 3) != (1, 2)

First differing element 0:
2
1

• (2, 3)
  + (1, 2)
  kwargs: {'bar': 'baz', 'foo': 'car'} != {'foo': 'bar', 'bar': 'baz'}
• {'bar': 'baz', 'foo': 'car'}
  ? ^

+ {'bar': 'baz', 'foo': 'bar'}
? ^

[cjw296]

This is a tricky one as there's plenty of prior art, with pytest's assertion rewriting [1], testfixtures compare [2] and the stuff that unittest already does [3].

I don't think any solution should rely on a TestCase being used as pytest, which is the most prevalent testing framework now, doesn't want to rely on them (they do come with a lot of baggage ;-)).

Can we make use of the pretty diffing stuff in unittest without explicitly making it a requirement?

[1] https://docs.pytest.org/en/latest/assert.html

[2] https://testfixtures.readthedocs.io/en/latest/comparing.html#dicts

[3] https://docs.python.org/3.8/library/unittest.html#unittest.TestCase.assertDictEqual

[Joshuah143]

Picking up this issue since I have spent a little too much of my time today sorting out issues with assert_called_with.

How would we feel about a system similar to this:

my_function('unexpected_input', 1, 1, 3, 4, 5)
mock_my_function.assert_called_with('expected_input', 1, 2, 3)

AssertionError: Expected call not found on call 2 of my_function.
Expected my_function(
        Arg(0) Expected: expected_input, Received: unexpected_input,
        ...
        Arg(2) Expected: 2, Received: 1,
        ...
)

my_function('expected_input', 1, 1, 3, bad_arg=True)
mock_my_function.assert_called_with('expected_input', 1, 2, 3)

AssertionError: Expected call not found on call 2 of my_function. Expected my_function with 0 kwargs, received 1 kwargs.
        Unexpected key "bad_arg" received in kwargs.

Assuming no opposition, I will open up a PR. The styling is somewhat custom, but chosen for simplicity.