zipfile: wrong encoding charset of member filename

[holymonson]

BPO	15602
Nosy	@loewis, @holymonson
Superseder	bpo-10614: ZipFile: add a filename_encoding argument

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2012-08-09.08:47:49.320>
created_at = <Date 2012-08-09.06:20:45.441>
labels = ['type-bug', 'library']
title = 'zipfile: wrong encoding charset of member filename'
updated_at = <Date 2012-08-09.08:47:49.319>
user = 'https://github.com/holymonson'

bugs.python.org fields:

activity = <Date 2012-08-09.08:47:49.319>
actor = 'loewis'
assignee = 'none'
closed = True
closed_date = <Date 2012-08-09.08:47:49.320>
closer = 'loewis'
components = ['Library (Lib)']
creation = <Date 2012-08-09.06:20:45.441>
creator = 'monson'
dependencies = []
files = []
hgrepos = []
issue_num = 15602
keywords = []
message_count = 2.0
messages = ['167760', '167775']
nosy_count = 2.0
nosy_names = ['loewis', 'monson']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = None
status = 'closed'
superseder = '10614'
type = 'behavior'
url = 'https://bugs.python.org/issue15602'
versions = ['Python 3.2']

[holymonson]

In /cpython/Lib/zipfile.py, there are some codes like

            if flags & 0x800:
                # UTF-8 file names extension
                filename = filename.decode('utf-8')
            else:
                # Historical ZIP filename encoding
                filename = filename.decode('cp437')

But actually there is no "Historical ZIP filename encoding", because zip files contain no charset info.
In English countries, it's usually not a big deal. But if the files zip on a non-cp437-based system (especially like China or Japan), filename is encoded from charsets like gb18030, but ZipFile decodes the byte stream to cp437, then everything goes wrong and people are hard to find the reason.

It's a problem new in py3k, and I found it on python3.2 and python3.4.
I suggest the filename returned in Bytes objects, or add decoding parameter when opening zipfile.

[loewis]

You are mistaken: there *is* a character set specification for file names in zip files, see

http://www.pkware.com/documents/casestudies/APPNOTE.TXT

Appendix D says

"The ZIP format has historically supported only the original IBM PC character encoding set, commonly referred to as IBM Code Page 437. This limits storing file name characters to only those within the original MS-DOS range of values and does not properly support file names in other character encodings, or languages."

Using bytes objects for file names is not acceptable; in Python 3, file names are (unicode) strings.

Adding a new parameter is an option, and already discussed in bpo-10614 .

People using non-437 code sets should really start using UTF-8 encoded file names in the zip files, and set the general purpose bit 11.

Closing this report as a duplicate.