MemoryError on zip.read in shutil._unpack_zipfile

[igorvoltaic]

BPO	43650
Nosy	@gpshead, @animalize, @miss-islington, @igorvoltaic
PRs	bpo-43650: Fix MemoryError on zip.read in shutil._unpack_zipfile for large files #25058 [3.10] bpo-43650: Fix MemoryError on zip.read in shutil._unpack_zipfile for large files (GH-25058) #26190 [3.9] bpo-43650: Fix MemoryError on zip.read in shutil._unpack_zipfile for large files (GH-25058) #26191

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/gpshead'
closed_at = <Date 2021-05-17.17:36:05.781>
created_at = <Date 2021-03-28.21:21:23.775>
labels = ['3.11', 'library', '3.9', '3.10', 'performance']
title = 'MemoryError on zip.read in shutil._unpack_zipfile'
updated_at = <Date 2021-05-17.17:36:05.780>
user = 'https://github.com/igorvoltaic'

bugs.python.org fields:

activity = <Date 2021-05-17.17:36:05.780>
actor = 'gregory.p.smith'
assignee = 'gregory.p.smith'
closed = True
closed_date = <Date 2021-05-17.17:36:05.781>
closer = 'gregory.p.smith'
components = ['Library (Lib)']
creation = <Date 2021-03-28.21:21:23.775>
creator = 'igorvoltaic'
dependencies = []
files = []
hgrepos = []
issue_num = 43650
keywords = ['patch']
message_count = 5.0
messages = ['389652', '393692', '393819', '393820', '393821']
nosy_count = 5.0
nosy_names = ['gregory.p.smith', 'python-dev', 'malin', 'miss-islington', 'igorvoltaic']
pr_nums = ['25058', '26190', '26191']
priority = 'normal'
resolution = 'fixed'
stage = 'commit review'
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue43650'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

[igorvoltaic]

MemoryError: null
...
File "....", line 13, in repack__file
shutil.unpack_archive(local_file_path, local_dir)
File "python3.6/shutil.py", line 983, in unpack_archive
func(filename, extract_dir, **kwargs)
File "python3.6/shutil.py", line 901, in _unpack_zipfile
data = zip.read(info.filename)
File "python3.6/zipfile.py", line 1338, in read
return fp.read()
File "python3.6/zipfile.py", line 858, in read
buf += self._read1(self.MAX_N)
File "python3.6/zipfile.py", line 948, in _read1
data = self._decompressor.decompress(data, n)

shutil.unpack_archive tries to read the whole file into memory, making use of any buffer at all. Python crashes for really large files. In my case — archive: ~1.7G, unpacked: ~10G. Interestingly zipfile.ZipFile.extractall handles this case more effective.

[igorvoltaic]

pls review

[miss-islington]

New changeset 049c412 by Miss Islington (bot) in branch '3.9':
bpo-43650: Fix MemoryError on zip.read in shutil._unpack_zipfile for large files (GH-25058)
049c412

[gpshead]

New changeset 7a58862 by Miss Islington (bot) in branch '3.10':
bpo-43650: Fix MemoryError on zip.read in shutil._unpack_zipfile for large files (GH-25058) (GH-26190)
7a58862

[gpshead]

thanks for the patch!