zipfile with multiprocessing: zipfile.BadZipFile

[maxime-lemonnier]

BPO	39363
Files	test_filesource.py: python script foo_bar_small.zip

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-01-16.18:24:07.296>
labels = ['library', 'type-crash']
title = 'zipfile with multiprocessing: zipfile.BadZipFile'
updated_at = <Date 2020-01-16.18:32:16.359>
user = 'https://bugs.python.org/maxime-lemonnier'

bugs.python.org fields:

activity = <Date 2020-01-16.18:32:16.359>
actor = 'maxime-lemonnier'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2020-01-16.18:24:07.296>
creator = 'maxime-lemonnier'
dependencies = []
files = ['48847', '48848']
hgrepos = []
issue_num = 39363
keywords = []
message_count = 2.0
messages = ['360134', '360135']
nosy_count = 1.0
nosy_names = ['maxime-lemonnier']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue39363'
versions = ['Python 3.6']

[maxime-lemonnier]

zipfile sometimes throws zipfile.BadZipFile when opening the same zip file from multiple processes

see attached file to reproduce the error. You'll need a zipfile with multiple files in it to reproduce.

[maxime-lemonnier]

Here's my console output:

python3 test_filesource.py
lock
file
access_mode = file, nb processes = 1, res = 110289, 0.08039402961730957 ms/frame
file
access_mode = file, nb processes = 4, res = 110289, 0.32297492027282715 ms/frame
lock
access_mode = lock, nb processes = 4, res = 110289, 0.2950408458709717 ms/frame
lock
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/path/to/script/test_filesource.py", line 64, in read_small
    return fs_small[i%len(fs_small)][42]
  File "/path/to/script/test_filesource.py", line 55, in __getitem__
    data_bytes = self.access( lambda archive: archive.read(member))
  File "/path/to/script/test_filesource.py", line 27, in access_lock
    return f(self.archive)
  File "/path/to/script/test_filesource.py", line 55, in <lambda>
    data_bytes = self.access( lambda archive: archive.read(member))
  File "/usr/lib/python3.6/zipfile.py", line 1337, in read
    with self.open(name, "r", pwd) as fp:
  File "/usr/lib/python3.6/zipfile.py", line 1419, in open
    % (zinfo.orig_filename, fname))
zipfile.BadZipFile: File name in directory '00000005.pkl' and header b'00000004.pkl' differ.
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/path/to/script/test_filesource.py", line 90, in <module>
    f(4, "lock") #crash
  File "/path/to/script/test_filesource.py", line 81, in f
    for i in pool.imap_unordered(read_small, frames):
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 735, in next
    raise value
zipfile.BadZipFile: File name in directory '00000005.pkl' and header b'00000004.pkl' differ.

[gpshead]

It appears you are attempting to use the same already opened zipfile instance from multiple different processes at once. This won't even work with multiprocessing start methods other than "fork" as the other processes would not inherit the open file descriptors and the zipfile instance would fail to pickle/unpickle to go to the other process.

In the "fork"ing case, an existing zipfile instance already has an open file and is thus not safe for use from those processes as they'll all be attempting to use the same underlying inherited open file descriptor at once.

zipfile is not going to try to prevent this.

Recommendation: Open a new zipfile instance from within each of the processes. parallel reading will work fine then as they no longer share any state.

Other recommendation: Do not use the "fork" multiprocessing start method. use "forkserver" or "spawn". you'll have far fewer problems. (the default start method on posix systems will change to one of those in the future)