Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multiprocessing not propagating -Xfrozen_modules=off #106723

Closed
ergoithz opened this issue Jul 13, 2023 · 0 comments · Fixed by #106724
Closed

Multiprocessing not propagating -Xfrozen_modules=off #106723

ergoithz opened this issue Jul 13, 2023 · 0 comments · Fixed by #106724
Assignees
Labels
3.11 only security fixes 3.12 bugs and security fixes topic-multiprocessing type-bug An unexpected behavior, bug, or error

Comments

@ergoithz
Copy link
Contributor

ergoithz commented Jul 13, 2023

Bug report

After python3.11 changes around frozen imports, when using multiprocessing contexts other than fork, the newly added -Xfrozen_modules=off isn't passed to spawned process interpreters.

Simple snippet demonstrating the issue:

"""

$ python -Xfrozen_modules=off test.py
main: {'frozen_modules': 'off'}
forkserver: {}
spawn: {}
fork: {'frozen_modules': 'off'}

"""

import sys
import multiprocessing

def xoptions():
    return sys._xoptions

def main():
    print('main:', xoptions())
    for ctx in ('forkserver', 'spawn', 'fork'):
        with multiprocessing.get_context(ctx).Pool(1) as pool:
            print(f'{ctx}:', pool.apply(xoptions))

if __name__ == '__main__':
    main()

The issue seems to be subprocess._args_from_interpreter_flags not honoring frozen_modules key from sys._xoptions.

$ python -Xfrozen_modules=off -c 'import subprocess;print(subprocess._args_from_interpreter_flags())'
[]

Your environment

python 3.11.4

Linked PRs

@ergoithz ergoithz added the type-bug An unexpected behavior, bug, or error label Jul 13, 2023
@gpshead gpshead self-assigned this Jul 28, 2023
@gpshead gpshead added 3.11 only security fixes 3.12 bugs and security fixes labels Jul 28, 2023
gpshead added a commit that referenced this issue Jul 28, 2023
…eters (#106724)

Co-authored-by: Kumar Aditya <kumaraditya@python.org>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Jul 28, 2023
…nterpreters (pythonGH-106724)

(cherry picked from commit 3dcac78)

Co-authored-by: Felipe A. Hernandez <ergoithz@gmail.com>
Co-authored-by: Kumar Aditya <kumaraditya@python.org>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
miss-islington pushed a commit to miss-islington/cpython that referenced this issue Jul 28, 2023
…nterpreters (pythonGH-106724)

(cherry picked from commit 3dcac78)

Co-authored-by: Felipe A. Hernandez <ergoithz@gmail.com>
Co-authored-by: Kumar Aditya <kumaraditya@python.org>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
kumaraditya303 added a commit that referenced this issue Jul 28, 2023
…interpreters (GH-106724) (#107368)

gh-106723: forward -Xfrozen_modules option to spawned process interpreters (GH-106724)
(cherry picked from commit 3dcac78)

Co-authored-by: Felipe A. Hernandez <ergoithz@gmail.com>
Co-authored-by: Kumar Aditya <kumaraditya@python.org>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
kumaraditya303 added a commit that referenced this issue Jul 28, 2023
…interpreters (GH-106724) (#107367)

gh-106723: forward -Xfrozen_modules option to spawned process interpreters (GH-106724)
(cherry picked from commit 3dcac78)

Co-authored-by: Felipe A. Hernandez <ergoithz@gmail.com>
Co-authored-by: Kumar Aditya <kumaraditya@python.org>
Co-authored-by: Gregory P. Smith <greg@krypto.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
3.11 only security fixes 3.12 bugs and security fixes topic-multiprocessing type-bug An unexpected behavior, bug, or error
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants