gzip fails to read a gzipped file (ValueError: readline of closed file)

[minstrelofc]

BPO	45475
Nosy	@methane, @miss-islington, @tirkarthi
PRs	bpo-45475: Revert __iter__ optimization for GzipFile, BZ2File, and LZMAFile. #29016 [3.10] bpo-45475: Revert __iter__ optimization for GzipFile, BZ2File, and LZMAFile. (GH-29016) #29050
Files	UTF-8-test_for_gzip.txt.gz: gzip test file, just in case

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-10-19.03:32:27.805>
created_at = <Date 2021-10-14.21:12:01.569>
labels = ['type-bug', 'library', '3.10', '3.11']
title = 'gzip fails to read a gzipped file (ValueError: readline of closed file)'
updated_at = <Date 2021-10-19.03:32:27.804>
user = 'https://bugs.python.org/minstrelofc'

bugs.python.org fields:

activity = <Date 2021-10-19.03:32:27.804>
actor = 'methane'
assignee = 'none'
closed = True
closed_date = <Date 2021-10-19.03:32:27.805>
closer = 'methane'
components = ['Library (Lib)']
creation = <Date 2021-10-14.21:12:01.569>
creator = 'minstrelofc'
dependencies = []
files = ['50358']
hgrepos = []
issue_num = 45475
keywords = ['patch', '3.10regression']
message_count = 5.0
messages = ['403948', '403981', '404061', '404263', '404265']
nosy_count = 4.0
nosy_names = ['methane', 'miss-islington', 'xtreak', 'minstrelofc']
pr_nums = ['29016', '29050']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue45475'
versions = ['Python 3.10', 'Python 3.11']

[minstrelofc]

Attempting to iterate over an opened gzip file raises a ValueError: readline of closed file

Behavior in Python 3.9.7:
Python 3.9.7 (default, Oct 13 2021, 09:08:19) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import gzip
>>> ll = [l for l in gzip.GzipFile(filename='data/UTF-8-test_for_gzip.txt.gz')]
>>> len(ll)
300


Behavior in Python 3.10.0 (and 3.11.0a1 is the same):
Python 3.10.0 (default, Oct 13 2021, 08:53:15) [GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import gzip
>>> ll = [l for l in gzip.GzipFile(filename='data/UTF-8-test_for_gzip.txt.gz')]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 1, in <listcomp>
ValueError: readline of closed file


This only happens when iterating directly over the GzipFile object. Using a with: statement has the correct behaviour in both 3.10 and 3.11:
>>> with gzip.GzipFile(filename='UTF-8-test_for_gzip.txt.gz') as input_file:
...     len(list(input_file))
... 
300

[tirkarthi]

This might be related to below commit :

commit d2a8e69
Author: Inada Naoki <songofacandy@gmail.com>
Date: Tue Apr 13 13:51:49 2021 +0900

bpo-43787: Add __iter__ to GzipFile, BZ2File, and LZMAFile (GH-25353)

python -m gzip README.rst
(myenv) ➜  cpython git:(main) ✗ git checkout d2a8e69c2c605fbaa3656a5f99aa8d295f74c80e~1 Lib/gzip.py
Updated 1 path from 2ea7c00ab4
(myenv) ➜  cpython git:(main) ✗ ./python
Python 3.11.0a1+ (heads/main:160c38df7f, Oct 15 2021, 11:25:16) [GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import gzip
>>> len([None for _ in gzip.GzipFile("README.rst.gz")])
267
>>> 
(myenv) ➜  cpython git:(main) ✗ git checkout d2a8e69c2c605fbaa3656a5f99aa8d295f74c80e Lib/gzip.py 
Updated 1 path from 1f9874eec6
(myenv) ➜  cpython git:(main) ✗ ./python
Python 3.11.0a1+ (heads/main:160c38df7f, Oct 15 2021, 11:25:16) [GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import gzip
>>> len([None for _ in gzip.GzipFile("README.rst.gz")])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 1, in <listcomp>
ValueError: readline of closed file

[methane]

>> ll = [l for l in gzip.GzipFile(filename='data/UTF-8-test_for_gzip.txt.gz')]

This is bad code pattern because you don't close the file explicitly.
Actually, the error caused by the optimization thet iter(GzipFile) returns underlaying faster iterator that don't have reference to the GzipFile. So GzipFile.__del__ close the file.

Although this is caused by bad code pattern, I must admit this is a regression.
We need to call slow Python function for each lines instead of using fast C iterator...

[methane]

New changeset 0a4c82d by Inada Naoki in branch 'main':
bpo-45475: Revert __iter__ optimization for GzipFile, BZ2File, and LZMAFile. (GH-29016)
0a4c82d

[miss-islington]

New changeset 97ce855 by Miss Islington (bot) in branch '3.10':
bpo-45475: Revert __iter__ optimization for GzipFile, BZ2File, and LZMAFile. (GH-29016)
97ce855