Opening a file holds the GIL when it calls "isatty()"

[smurfix]

BPO	44219
Nosy	@vstinner, @smurfix, @ambv, @vxgmichel, @miss-islington
PRs	bpo-44219: Release the GIL during isatty syscalls #28250 [3.10] bpo-44219: Release the GIL during isatty syscalls (GH-28250) #28255 [3.9] bpo-44219: Release the GIL during isatty syscalls (GH-28250) #28256 bpo-44219: Mention GH-28250 is a deadlock bugfix #28261 [3.10] bpo-44219: Mention GH-28250 is a deadlock bugfix (GH-28261) #28274 [3.9] bpo-44219: Mention GH-28250 is a deadlock bugfix (GH-28261) #28275
Files	bpo-44219.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-09-09.16:42:10.377>
created_at = <Date 2021-05-23.17:18:13.788>
labels = ['interpreter-core', '3.10', '3.9', 'type-crash', '3.11']
title = 'Opening a file holds the GIL when it calls "isatty()"'
updated_at = <Date 2021-09-10.16:22:31.565>
user = 'https://github.com/smurfix'

bugs.python.org fields:

activity = <Date 2021-09-10.16:22:31.565>
actor = 'lukasz.langa'
assignee = 'none'
closed = True
closed_date = <Date 2021-09-09.16:42:10.377>
closer = 'vstinner'
components = ['Interpreter Core']
creation = <Date 2021-05-23.17:18:13.788>
creator = 'smurfix'
dependencies = []
files = ['50270']
hgrepos = []
issue_num = 44219
keywords = ['patch']
message_count = 12.0
messages = ['394208', '401261', '401402', '401450', '401458', '401470', '401478', '401495', '401497', '401584', '401588', '401591']
nosy_count = 5.0
nosy_names = ['vstinner', 'smurfix', 'lukasz.langa', 'vxgmichel', 'miss-islington']
pr_nums = ['28250', '28255', '28256', '28261', '28274', '28275']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue44219'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

[smurfix]

Opening a file calls isatty which calls an ioctl with the GIL held.

GDB:

#0  __GI___tcgetattr (fd=18, termios_p=termios_p@entry=0x7f618a5df920)
    at ../sysdeps/unix/sysv/linux/tcgetattr.c:38
#1  0x00007f618bd1ca0c in __isatty (fd=<optimized out>) at ../sysdeps/posix/isatty.c:27
#2  0x000000000062b746 in _Py_device_encoding (fd=<optimized out>) at ../Python/fileutils.c:62
#3  0x000000000060bf90 in _io_TextIOWrapper___init___impl (write_through=0, line_buffering=0, 
    newline=0x0, errors='strict', encoding=<optimized out>, 
    buffer=<_io.BufferedWriter at remote 0x7f618986aeb0>, self=0x7f618985a860)
    at ../Modules/_io/textio.c:1149
...

Please don't do that.

In my case, the file in question is implemented as a FUSE mount which is served by the same process (different thread of course). Thus holding the GIL at this point causes a rather interesting deadlock.

Tested with 3.9.

[vxgmichel]

My team ran into this issue while developing a fuse application too.

In an effort to help this issue move forward, I tried to list all occurrences of the isatty C function in the cpython code base. I found 14 of them.

9 of them are directly related to stdin/stdout/stderr, so it's probably not crucial to release the GIL for those occurrences:

• main.c:stdin_is_interactive
• main.c:pymain_import_readline
• readline.c:setup_readline
• bltinmodule.c:builtin_input_impl (x2)
• frozenmain.c:Py_FrozenMain
• pylifecycle.c:Py_FdIsInteractive (x2)
• fileobject.c:stdprinter_isatty (GIL is actually released for this one)

Out of the remaining 4, only 1 releases the GIL:

• fileio.c:_io_FileIO_isatty_impl: used for FileIO.isatty

Which gives 3 occurrences of non-stdstream specific usage of isatty that do not release the GIL:

• posixmodule.c:os_isatty_impl: used by os.isatty
• fileutils.c:_Py_device_encoding: used TextIOWrapper.__init__
• fileutils.c:_Py_write_impl: windows specific, issue bpo-11395

The first one is used by os.isatty which means this call can also deadlock. I did manage to reproduce it with a simple fuse loopback file system: https://github.com/fusepy/fusepy/blob/master/examples/loopback.py

The second one is the one found by @smurfix and gets triggered when io.open() is used in text mode.

The third one only triggers on windows when writing more than 32767 bytes to a file descriptor. A comment points to issue bpo-11395 (https://bugs.python.org/issue11395). Also, it seems from the function signature that this function might be called with or without the GIL held, which might cause the fix to be a bit more complicated than the first two use cases.

I hope this helps.

[vxgmichel]

Here's a possible patch that fixes the 3 unprotected calls to isatty mentioned above. It successfully passes the test suite. I can submit a PR with this patch if necessary.

[smurfix]

Please do. However, I do think that changing the stdstream related ioctl calls also is a good idea, if only for code regularity/completeness sake.

(Besides, nothing prevents somebody from starting a FUSE file system and then redirecting stdout to it …)

[vxgmichel]

There are a couple of reasons why I did not make changes to the stdstream related functions.

The first one is that a PR with many changes is less likely to get reviewed and merged than a PR with fewer changes. The second one is that it's hard for me to make sure that those functions are always called with the GIL already held. Maybe some of them never hold the GIL in the first place, and I'm not familiar enough with the code base to tell.

So in the end it will probably be up to the coredev reviewing the PR, but better start small.

Besides, nothing prevents somebody from starting a FUSE file system and then redirecting stdout to it …

I ran some checks and python -c 'input()' < my-fuse-mountpoint/data_in does indeed trigger an ioctl call to the corresponding fuse file system. But how would my-fuse-mountpoint/data_in be the stdin for the process that itself starts the fuse filesystem? I don't see how to generate this deadlock, apart from setting up interprocess communication between the fuse process and the my-fuse-mountpoint/data_in-as-stdin process.

[vstinner]

New changeset 06148b1 by Vincent Michel in branch 'main':
bpo-44219: Release the GIL during isatty syscalls (GH-28250)
06148b1

[miss-islington]

New changeset 5c65834 by Miss Islington (bot) in branch '3.9':
bpo-44219: Release the GIL during isatty syscalls (GH-28250)
5c65834

[ambv]

New changeset 23c4677 by Miss Islington (bot) in branch '3.10':
bpo-44219: Release the GIL during isatty syscalls (GH-28250) (GH-28255)
23c4677

[vstinner]

In my case, the file in question is implemented as a FUSE mount which is served by the same process (different thread of course). Thus holding the GIL at this point causes a rather interesting deadlock.

Since the change fixes a deadlock, I agree to backport it to 3.9 and 3.10.

Thanks for the fix!

[ambv]

New changeset 296b710 by Łukasz Langa in branch 'main':
bpo-44219: Mention #72437 is a deadlock bugfix (GH-28261)
296b710

[ambv]

New changeset 60ddf49 by Miss Islington (bot) in branch '3.10':
bpo-44219: Mention #72437 is a deadlock bugfix (GH-28261) (GH-28274)
60ddf49

[ambv]

New changeset 314de53 by Miss Islington (bot) in branch '3.9':
bpo-44219: Mention #72437 is a deadlock bugfix (GH-28261) (GH-28275)
314de53