MagicMock specialisation instance can no longer be passed to new MagicMock instance

[FrankHarrison]

BPO	39578
Nosy	@cjw296, @voidspace, @elenaoat, @lisroach, @mariocj89, @tirkarthi

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-02-07.09:38:41.606>
labels = ['3.8', 'library', '3.9']
title = 'MagicMock specialisation instance can no longer be passed to new MagicMock instance'
updated_at = <Date 2020-02-11.00:03:09.479>
user = 'https://bugs.python.org/FrankHarrison'

bugs.python.org fields:

activity = <Date 2020-02-11.00:03:09.479>
actor = 'Elena.Oat'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2020-02-07.09:38:41.606>
creator = 'Frank Harrison'
dependencies = []
files = []
hgrepos = []
issue_num = 39578
keywords = []
message_count = 5.0
messages = ['361552', '361554', '361657', '361723', '361763']
nosy_count = 7.0
nosy_names = ['cjw296', 'michael.foord', 'Elena.Oat', 'lisroach', 'mariocj89', 'xtreak', 'Frank Harrison']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue39578'
versions = ['Python 3.8', 'Python 3.9']

[FrankHarrison]

This is my first bug logged here, I've tried to follow the guideline and search for this issue; please let me know if I missed anything.

Summary:
unittest.mock.MagicMock has a regression starting in 3.8. The regression was only tested on latest non-prerelease versions of python 3.5, 3.6, 3.7, 3.8 and 3.9. Tested on OSX and Fedora 31.

Repro:
------
If you create an instance of a MagicMock specialisation with parameters to __init__(), you can no longer pass that instance to the __init__() function of another MagicMock object e.g. a base-class is replaced with MagicMock. See the unittests bellow for more details, use-cases and fail situations.

What happens:
-------------
Here's a python3.9 example traceback. It may be worth noting that there is a difference in the tracebacks between 3.8 and 3.9.

Traceback (most recent call last):
  File "<...>", line <..>, in test_raw_magic_moc_passing_thru_single_pos
    mock_object = mock.MagicMock(mock_param)  # error is here, instantiating another object
  File "/usr/lib64/python3.9/unittest/mock.py", line 408, in __new__
    if spec_arg and _is_async_obj(spec_arg):
  File "/usr/lib64/python3.9/unittest/mock.py", line 2119, in __get__
    return self.create_mock()
  File "/usr/lib64/python3.9/unittest/mock.py", line 2112, in create_mock
    m = parent._get_child_mock(name=entry, _new_name=entry,
  File "/usr/lib64/python3.9/unittest/mock.py", line 1014, in _get_child_mock
    return klass(**kw)
TypeError: __init__() got an unexpected keyword argument 'name'

Code demonstrating the problem:
-------------------------------

import unittest

from unittest import mock


class TestMockMagicAssociativeHierarchies(unittest.TestCase):
    """ Mimicing real-world testing where we mock a base-class

The intent here is to demonstrate some of the requirements of associative-
hierarchies e.g. where a class may have its associative-parent set at
run-time, rather that defining it via a class-hierarchy. Obviously this
also needs to work with class-hierarchies, that is an associative-parent is
likely to be a specialisation of some interface, usually one that is being
mocked.

For example tkinter and Qt have both a class-hierarchy and a layout-
hierarchy; the latter is modifyable at runtime.

Most of the tests here mimic a specialisation of an upstream object (say a
tk.Frame class), instantiating that specialisation and then passing it to
another object. The reason behind this is an observed regression in Python
3.8.
"""
def test_raw_magic_moc_passing_thru_no_params(self):
    """ REGRESSION: Python3.8 (inc Python3.9)

    Create a mocked specialisation passing it to another mock.

    One real-world use-case for this is simple cases where we simply want to
    define a new convenience type that forces a default configuration of
    the inherited type (calls super().__init__()).
    """
    class MockSubCallsParentInit(mock.MagicMock):
        def __init__(self):
            super().__init__()  # intentionally empty
    mock_param = MockSubCallsParentInit()
    mock_object = mock.MagicMock(mock_param)  # error is here, instantiating another object
    self.assertIsInstance(mock_object, mock.MagicMock)

    def test_raw_magic_moc_passing_thru_single_pos(self):
        """ REGRESSION: Python3.8 (inc Python3.9)

    Same as test_raw_magic_moc_no_init_params() but we want to specialise
    with positional arguments. """
    class MockSubCallsParentInitWithPositionalParam(mock.MagicMock):
        def __init__(self):
            super().__init__("specialise init calls")
    mock_param = MockSubCallsParentInitWithPositionalParam()
    mock_object = mock.MagicMock(mock_param)  # error is here, instantiating another object
    self.assertIsInstance(mock_object, mock.MagicMock)

    def test_raw_magic_moc_passing_thru_single_kwarg(self):
        """ REGRESSION: Python3.8 (inc Python3.9)

    Same as test_raw_magic_moc_passing_thru_single_pos() but we want to
    specialise with a key-word argument. """
    class MockSubCallsParentInitWithPositionalParam(mock.MagicMock):
        def __init__(self):
            super().__init__(__some_key_word__="some data")
    mock_param = MockSubCallsParentInitWithPositionalParam()
    mock_object = mock.MagicMock(mock_param)  # error is here, instantiating another object
    self.assertIsInstance(mock_object, mock.MagicMock)

    def test_mock_as_param_no_inheritance(self):
        """ PASSES 

    Mimic mocking out types, without type specialisation.
    for example in pseudo code 
        tk.Frame = mock.MagicMock; tk.Frame(t.Frame) """
    mock_param = mock.MagicMock()
    mock_object = mock.MagicMock(mock_param)
    self.assertIsInstance(mock_object, mock.MagicMock)

    def test_mock_as_param_no_init_override(self):
        """ PASSES 

    Leaves the __init__() function behaviour as default; should always
    work. Note that we do not specialise member functions.

    Although the intent here is similar to the one captured by
    test_raw_magic_moc_passing_thru_no_params(), this is a less likely
    usecase, although it does happen, but is here for completeness """
    class MockSub(mock.MagicMock): pass
    mock_param = MockSub()
    mock_object = mock.MagicMock(mock_param)
    self.assertIsInstance(mock_object, mock.MagicMock)

    def test_init_with_args_n_kwargs_passthru(self):
        """ PASSES

    Intended to be the same as test_mock_as_param_no_init_override as well
    as a base-test for ithe usecases where a user will define more complex
    behaviours such as key-word modification, member-variable definitions
    and so on. """
    class MockSubInitPassThruArgsNKwargs(mock.MagicMock):
        def __init__(self, *args, **kwargs):
            super().__init__(*args, **kwargs)  # intentionally redundant
    mock_param = MockSubInitPassThruArgsNKwargs()
    mock_object = mock.MagicMock(mock_param)
    self.assertIsInstance(mock_object, mock.MagicMock)

    def test_init_with_args_n_kwargs_modify_kwargs(self):
        """ PASSES

    Same as test_init_with_args_n_kwargs_passthru() but modifies the kwargs
    dict on the way through the __init__() function.
    """
    class MockSubModifyKwargs(mock.MagicMock):
        def __init__(self, *args, **kwargs):
            kwargs["__kw args added__"] = "test value"
            super().__init__(*args, **kwargs)
    mock_param = MockSubModifyKwargs()
    mock_object = mock.MagicMock(mock_param)
    self.assertIsInstance(mock_object, mock.MagicMock)

    def test_init_with_args_n_kwargs_modify_args(self):
        """ PASSES

    Same as test_init_with_args_n_kwargs_passthru() but modifies the args
    on their way through the __init__() function.
    """
    class MockSubModifyArgs(mock.MagicMock):
        def __init__(self, *args, **kwargs):
            super().__init__("test value", *args, **kwargs)
    mock_param = MockSubModifyArgs()
    mock_object = mock.MagicMock(mock_param)
    self.assertIsInstance(mock_object, mock.MagicMock)

[FrankHarrison]

Minor correction: The regression was only tested on Python 3.9.0a2 (Fedora), Python 3.9a3 (OSX), CPython's master (build from source) and the latest non-prerelease versions of python 3.5, 3.6, 3.7, and 3.8. Tested on OSX and Fedora 31.

[mariocj89]

Having not looked deeply at it but with the reproducer, running a quick bisect, it points to this commit: 77b3b77
I'll try having a look later at the day if I can manage to free some time.

[elenaoat]

I am looking at reproducing this and creating a short example of where this fails. Will look further into more details of why this doesn't work.

[elenaoat]

Here's the example I ran, that indeed fails in Python 3.8 and Python 3.9 (with different errors) and works in Python 3.7.

from unittest.mock import MagicMock


class CustomMock(MagicMock):
    def __init__(self):
        super().__init__(__something__='something')


mock = CustomMock()
MagicMock(mock)

In Python 3.8 the error is TypeError: __init__() got an unexpected keyword argument '_new_parent'.

In Python 3.9 the error is TypeError: __init__() got an unexpected keyword argument 'name'.