Add _at_fork_reinit() method to locks

[vstinner]

BPO	40089
Nosy	@vstinner, @miss-islington
PRs	bpo-40089: Fix threading._after_fork() #19191 [3.7] bpo-40089: Fix threading._after_fork() (GH-19191) #19193 [3.8] bpo-40089: Fix threading._after_fork() (GH-19191) #19194 bpo-40089: Add _at_fork_reinit() method to locks #19195 [WIP] bpo-40091: Fix a crash in logging after fork #19196

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-04-21.01:07:34.430>
created_at = <Date 2020-03-27.15:51:37.958>
labels = ['library', '3.9']
title = 'Add _at_fork_reinit() method to locks'
updated_at = <Date 2020-04-21.01:07:34.430>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2020-04-21.01:07:34.430>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2020-04-21.01:07:34.430>
closer = 'vstinner'
components = ['Library (Lib)']
creation = <Date 2020-03-27.15:51:37.958>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 40089
keywords = ['patch']
message_count = 8.0
messages = ['365157', '365165', '365168', '365171', '365176', '365945', '365947', '365948']
nosy_count = 3.0
nosy_names = ['vstinner', 'miss-islington', 'Batuhan Taskaya']
pr_nums = ['19191', '19193', '19194', '19195', '19196']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue40089'
versions = ['Python 3.9']

[vstinner]

Using a lock after fork() is unsafe and can crash.

Example of a crash in logging after a fork on AIX:
https://bugs.python.org/issue40068#msg365028

This problem is explained in length in bpo-6721: "Locks in the standard library should be sanitized on fork".

The threading module registers an "at fork" callback: Thread._reset_internal_locks() is called to reset self._started (threading.Event) and self._tstate_lock. The current implementation creates new Python lock objects and forgets about the old ones.

I propose to add a new _at_fork_reinit() method to Python lock objects which reinitializes the native lock internally without having to create a new Python object.

Currently, my implementation basically creates a new native lock object and forgets about the old new (don't call PyThread_free_lock()).

Tomorrow, we can imagine a more efficient impementation using platform specific function to handle this case without having to forget about the old lock.

[vstinner]

_at_fork() has a bug: it creates a _DummyThread instead of a _MainThread. Example:
---

import os, _thread, threading, time

def fork_in_thread():
    pid = os.fork()
    if pid:
        # parent
        os._exit(0)

# child process
print(f"child process: {threading.current_thread()=}")
print(f"child process: {threading._main_thread=}")

print(f"parent process: {threading.current_thread()=}")
print(f"parent process: {threading._main_thread=}")

_thread.start_new_thread(fork_in_thread, ())
# block the main thread: fork_in_thread() exit the process
time.sleep(60)

---

Output:
---
parent process: threading.current_thread()=<_MainThread(MainThread, started 139879200077632)>
parent process: threading._main_thread=<_MainThread(MainThread, started 139879200077632)>
child process: threading.current_thread()=<_DummyThread(Dummy-1, started daemon 139878980245248)>
child process: threading._main_thread=<_DummyThread(Dummy-1, started daemon 139878980245248)>
---

My PR 19191 fix the issue:
---
parent process: threading.current_thread()=<_MainThread(MainThread, started 140583665170240)>
parent process: threading._main_thread=<_MainThread(MainThread, started 140583665170240)>
child process: threading.current_thread()=<_MainThread(MainThread, started 140583445075712)>
child process: threading._main_thread=<_MainThread(MainThread, started 140583445075712)>
---

[vstinner]

New changeset d8ff44c by Victor Stinner in branch 'master':
bpo-40089: Fix threading._after_fork() (GH-19191)
d8ff44c

[vstinner]

See also bpo-40091 "Crash in logging._after_at_fork_child_reinit_locks()" that I just created.

[vstinner]

See also bpo-40092: "Crash in _PyThreadState_DeleteExcept() at fork in the process child".

[vstinner]

New changeset 87255be by Victor Stinner in branch 'master':
bpo-40089: Add _at_fork_reinit() method to locks (GH-19195)
87255be

[vstinner]

New changeset 6318e45 by Miss Islington (bot) in branch '3.8':
bpo-40089: Fix threading._after_fork() (GH-19191) (GH-19194)
6318e45

[vstinner]

New changeset a514ccb by Miss Islington (bot) in branch '3.7':
bpo-40089: Fix threading._after_fork() (GH-19191) (GH-19193)
a514ccb