Disallow fork in a subinterpreter.

[ericsnowcurrently]

BPO	34651
Nosy	@gpshead, @vstinner, @ericsnowcurrently, @hroncok, @miss-islington
PRs	bpo-34651: Only allow the main interpreter to fork. #9279 bpo-38778: Document that os.fork is not allowed in subinterpreters #17123 [3.8] bpo-38778: Document that os.fork is not allowed in subinterpreters (GH-17123) #17179

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ericsnowcurrently'
closed_at = <Date 2019-08-29.16:38:15.524>
created_at = <Date 2018-09-12.21:04:23.258>
labels = ['interpreter-core', '3.8']
title = 'Disallow fork in a subinterpreter.'
updated_at = <Date 2019-11-15.21:37:30.285>
user = 'https://github.com/ericsnowcurrently'

bugs.python.org fields:

activity = <Date 2019-11-15.21:37:30.285>
actor = 'miss-islington'
assignee = 'eric.snow'
closed = True
closed_date = <Date 2019-08-29.16:38:15.524>
closer = 'vstinner'
components = ['Interpreter Core']
creation = <Date 2018-09-12.21:04:23.258>
creator = 'eric.snow'
dependencies = []
files = []
hgrepos = []
issue_num = 34651
keywords = ['patch']
message_count = 12.0
messages = ['325181', '325302', '325400', '343364', '343369', '343415', '343421', '345599', '345603', '350652', '356694', '356720']
nosy_count = 5.0
nosy_names = ['gregory.p.smith', 'vstinner', 'eric.snow', 'hroncok', 'miss-islington']
pr_nums = ['9279', '17123', '17179']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue34651'
versions = ['Python 3.8']

[ericsnowcurrently]

os.fork() potentially has some problematic behavior when called from a subinterpreter. In additional to the normal fork+threads madness, there's the question of what to do with existing subinterpreters. The simplest solution is to simply disallow fork in a subinterpreter and then wipe out all subinterpreters in the child process post-fork (if os.fork() called in the main interpreter).

[gpshead]

+1 agreed. this is the simplest approach to start with.

Code to restrict: os.fork itself and disallowing the use of preexec_fn on subprocess within subinterpreters.

feel free to ignore preexec_fn in subprocess for the time being if desired, we already promise people that it is a bad idea legacy API that we don't want, guaranteed to cause undiagnosable problems and deadlocks at times. :)

[ericsnowcurrently]

New changeset 5903296 by Eric Snow in branch 'master':
bpo-34651: Only allow the main interpreter to fork. (gh-9279)
5903296

[hroncok]

It appears that as a result of this, subprocess.Popen cannot be called from within a subinterpreter either. Is that an obvious and desired limitation?

[vstinner]

I reopen the issue to let Eric answer ;-)

If the behavior is deliberate, maybe it should just be documented somewhere?

[gpshead]

I'd start by documenting the limitation and keeping a future feature request of "Allow subprocess to work from subinterpreters". That is doable. Supporting os.fork() is not.

[vstinner]

I'd start by documenting the limitation and keeping a future feature request of "Allow subprocess to work from subinterpreters". That is doable. Supporting os.fork() is not.

subprocess is able to use os.posix_spawn() in Python 3.8. In that case, it's not limited by os.fork() check ;-) Windows isn't limited neither, CreateProcess() isn't limited in subinterpreters.

[ericsnowcurrently]

FYI, I plan on looking into this either today or next Friday.

[vstinner]

See also bpo-37266: "Daemon threads must be forbidden in subinterpreters".

[hroncok]

The problem with subprocess.Popen has been fixed in https://bugs.python.org/issue37951