codecs.open leaks file descriptor when invalid encoding is passed

[BrockMendel]

BPO	38971
Nosy	@serhiy-storchaka, @MojoVampire, @miss-islington, @tirkarthi, @caporta
PRs	bpo-38971: open file in codecs.open() closes if exception raised #17666 [3.7] bpo-38971: Open file in codecs.open() closes if exception raised. (GH-17666) #18733 [3.8] bpo-38971: Open file in codecs.open() closes if exception raised. (GH-17666) #18734

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-03-02.08:16:43.886>
created_at = <Date 2019-12-04.16:51:47.592>
labels = ['easy', '3.8', '3.9', 'expert-IO', 'performance', '3.7', 'library']
title = 'codecs.open leaks file descriptor when invalid encoding is passed'
updated_at = <Date 2020-03-02.08:16:43.885>
user = 'https://bugs.python.org/BrockMendel'

bugs.python.org fields:

activity = <Date 2020-03-02.08:16:43.885>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = True
closed_date = <Date 2020-03-02.08:16:43.886>
closer = 'serhiy.storchaka'
components = ['Library (Lib)', 'IO']
creation = <Date 2019-12-04.16:51:47.592>
creator = 'Brock Mendel'
dependencies = []
files = []
hgrepos = []
issue_num = 38971
keywords = ['patch', 'easy']
message_count = 12.0
messages = ['357811', '357812', '357814', '357815', '357818', '357834', '357838', '360731', '363128', '363136', '363137', '363138']
nosy_count = 6.0
nosy_names = ['serhiy.storchaka', 'josh.r', 'miss-islington', 'xtreak', 'Brock Mendel', 'caporta']
pr_nums = ['17666', '18733', '18734']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue38971'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9']

[BrockMendel]

xref pandas-dev/pandas#30034

codecs.open does file = open(...) before validating the encoding kwarg, leaving the open file behind if that validation raises.

[tirkarthi]

Does using with block similar to https://bugs.python.org/issue22831 solve this problem?

[BrockMendel]

Does using with block similar to https://bugs.python.org/issue22831 solve this problem?

The motivating use case uses `with codecs.open(buf, "w", encoding=encoding) as f:`

https://github.com/pandas-dev/pandas/blob/master/pandas/io/formats/format.py#L498

[tirkarthi]

Ah okay, thanks for the detail. Forgot there should be an open file handle to be returned by codecs.open

[serhiy-storchaka]

Add

try:
    ...
except:
    file.close()
    raise

[MojoVampire]

Any reason not to just defer opening the file until after the codec has been validated, so the resource acquisition comes last?

[serhiy-storchaka]

Many reasons.

1. It is simpler.
2. We will need a try/except in any case to prevent a leak if an exception be raised by other code following open().
3. It matches the behavior of io.open().

[caporta]

Just quickly pinging the thread as a friendly reminder that PR 17666 is open and potentially close to mergeable, as it's been through two review cycles already (thanks Serhiy). If someone has the bandwidth to take another look, it would be greatly appreciated. Thanks!

[serhiy-storchaka]

New changeset 2565ede by Chris A in branch 'master':
bpo-38971: Open file in codecs.open() closes if exception raised. (GH-17666)
2565ede

[miss-islington]

New changeset f4d709f by Miss Islington (bot) in branch '3.7':
bpo-38971: Open file in codecs.open() closes if exception raised. (GH-17666)
f4d709f

[miss-islington]

New changeset f28b0c7 by Miss Islington (bot) in branch '3.8':
bpo-38971: Open file in codecs.open() closes if exception raised. (GH-17666)
f28b0c7

[serhiy-storchaka]

It is too later for 2.7.