Importing multiprocessing breaks pickle.whichmodule

[renatolfc]

BPO	42406
Nosy	@gpshead, @pitrou, @renatolfc, @applio, @miss-islington
PRs	bpo-42406: Fix whichmodule() with multiprocessing #23403 [3.9] bpo-42406: Fix whichmodule() with multiprocessing (GH-23403) #23560 [3.8] bpo-42406: Fix whichmodule() with multiprocessing (GH-23403) #23561 [3.7] bpo-42406: Fix whichmodule() with multiprocessing (pythonGH-23403) #28489

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/gpshead'
closed_at = <Date 2020-11-29.23:48:33.708>
created_at = <Date 2020-11-19.13:19:31.708>
labels = ['3.8', 'type-bug', 'library', '3.9', '3.10']
title = 'Importing multiprocessing breaks pickle.whichmodule'
updated_at = <Date 2021-09-21.11:17:37.801>
user = 'https://github.com/renatolfc'

bugs.python.org fields:

activity = <Date 2021-09-21.11:17:37.801>
actor = 'python-dev'
assignee = 'gregory.p.smith'
closed = True
closed_date = <Date 2020-11-29.23:48:33.708>
closer = 'gregory.p.smith'
components = ['Library (Lib)']
creation = <Date 2020-11-19.13:19:31.708>
creator = 'renatolfc'
dependencies = []
files = []
hgrepos = []
issue_num = 42406
keywords = ['patch']
message_count = 7.0
messages = ['381410', '382024', '382085', '382086', '382088', '382097', '382160']
nosy_count = 6.0
nosy_names = ['gregory.p.smith', 'pitrou', 'renatolfc', 'python-dev', 'davin', 'miss-islington']
pr_nums = ['23403', '23560', '23561', '28489']
priority = 'normal'
resolution = 'fixed'
stage = 'commit review'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue42406'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

[renatolfc]

Importing multiprocessing prior to other modules that define ufuncs breaks pickle.whichmodule for those functions.

Example:

    Python 3.8.6 (default, Sep 30 2020, 04:00:38)
    [GCC 10.2.0] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import multiprocessing
    >>> from scipy.special import gdtrix
    >>> from pickle import whichmodule
    >>> whichmodule(gdtrix, gdtrix.__name__)
    '__mp_main__'

Now, if we change the order of imports, whichmodule works fine:

    Python 3.8.6 (default, Sep 30 2020, 04:00:38)
    [GCC 10.2.0] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> from scipy.special import gdtrix
    >>> from pickle import whichmodule
    >>> import multiprocessing
    >>> whichmodule(gdtrix, gdtrix.__name__)
    'scipy.special._ufuncs'

This happens because multiprocessing creates an alias to __main__ when imported:

[Lib/multiprocessing/__init__.py](https://github.com/python/cpython/blob/main/Lib/multiprocessing/__init__.py)
37:    sys.modules['__mp_main__'] = sys.modules['__main__']

But doing so changes the ordering of sys.modules, and whichmodule returns __mp_main__ instead of the correct scipy.special._ufuncs.

This bug has arisen in the context of using dill (which imports multiprocessing) for serialization and is further documented at uqfoundation/dill#392.

I intend to submit a patch/PR with a fix soon.

[gpshead]

gross / nice find :)

[gpshead]

New changeset 8668431 by Renato Cunha in branch 'master':
bpo-42406: Fix whichmodule() with multiprocessing (GH-23403)
8668431

[miss-islington]

New changeset b1c48e5 by Miss Islington (bot) in branch '3.8':
bpo-42406: Fix whichmodule() with multiprocessing (GH-23403)
b1c48e5

[miss-islington]

New changeset fcf7391 by Miss Islington (bot) in branch '3.9':
bpo-42406: Fix whichmodule() with multiprocessing (GH-23403)
fcf7391

[gpshead]

thanks Renato!

[renatolfc]

No worries, Gregory. Glad I could help. :-)