Possible resource warning in "with open()"

[serhiy-storchaka]

BPO	34066
Nosy	@ncoghlan, @benjaminp, @jwilk, @markshannon, @serhiy-storchaka, @1st1, @miss-islington
PRs	bpo-34066: Disabled interruption before SETUP_WITH and BEFORE_ASYNC_WITH. #8159 [3.7] bpo-34066: Disabled interruption before SETUP_WITH and BEFORE_ASYNC_WITH. (GH-8159) #8197 [3.6] bpo-34066: Disabled interruption before SETUP_WITH and BEFORE_ASYNC_WITH. (GH-8159) #8198

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-07-09.16:03:41.491>
created_at = <Date 2018-07-07.17:11:52.350>
labels = ['interpreter-core', '3.7', '3.8', 'performance']
title = 'Possible resource warning in "with open()"'
updated_at = <Date 2018-07-13.16:19:58.631>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2018-07-13.16:19:58.631>
actor = 'jwilk'
assignee = 'none'
closed = True
closed_date = <Date 2018-07-09.16:03:41.491>
closer = 'serhiy.storchaka'
components = ['Interpreter Core']
creation = <Date 2018-07-07.17:11:52.350>
creator = 'serhiy.storchaka'
dependencies = []
files = []
hgrepos = []
issue_num = 34066
keywords = ['patch']
message_count = 5.0
messages = ['321224', '321288', '321314', '321316', '321328']
nosy_count = 7.0
nosy_names = ['ncoghlan', 'benjamin.peterson', 'jwilk', 'Mark.Shannon', 'serhiy.storchaka', 'yselivanov', 'miss-islington']
pr_nums = ['8159', '8197', '8198']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue34066'
versions = ['Python 3.6', 'Python 3.7', 'Python 3.8']

[serhiy-storchaka]

The bytecode generated for "with open()":

with open(path) as file:
    data = file.read()

1 0 LOAD_NAME 0 (open)
2 LOAD_NAME 1 (path)
4 CALL_FUNCTION 1
6 SETUP_WITH 14 (to 22)
8 STORE_NAME 2 (file)

2 10 LOAD_NAME 2 (file)
12 LOAD_METHOD 3 (read)
14 CALL_METHOD 0
16 STORE_NAME 4 (data)
18 POP_BLOCK
20 BEGIN_FINALLY
>> 22 WITH_CLEANUP_START
24 WITH_CLEANUP_FINISH
26 END_FINALLY
28 LOAD_CONST 0 (None)
30 RETURN_VALUE

The execution can be interrupted by Ctrl-C between calling open() and entering the 'with' block. In this case the file object will be created, but its __enter__ and __exit__ methods will be not executed. As a result it will be closed after disappearing a reference to it and a ResourceWarning will be emitted.

The solution is disabling interruption before the SETUP_WITH opcode. It is already disabled before SETUP_FINALLY and YIELD_FROM. It is worth to disable it before BEFORE_ASYNC_WITH for consistency although I don't have examples for it.

See also bpo-29988.

[serhiy-storchaka]

Maybe it is worth to disable interrupting before more opcodes. For example for fixing a problem with the contextlib.nullcontext example (bpo-34067) we need to disable interrupting before STORE_FAST, LOAD_FAST and JUMP_FORWARD.

[ncoghlan]

New changeset 3f4d90d by Nick Coghlan (Serhiy Storchaka) in branch 'master':
bpo-34066: Disabled interruption before SETUP_WITH and BEFORE_ASYNC_WITH. (GH-8159)
3f4d90d

[miss-islington]

New changeset f5197dd by Miss Islington (bot) in branch '3.7':
bpo-34066: Disabled interruption before SETUP_WITH and BEFORE_ASYNC_WITH. (GH-8159)
f5197dd

[serhiy-storchaka]

New changeset eeaae26 by Serhiy Storchaka in branch '3.6':
[3.6] bpo-34066: Disabled interruption before SETUP_WITH and BEFORE_ASYNC_WITH. (GH-8159) (GH-8198)
eeaae26