Python 2.7.4 breaks ZipFile extraction of zip files with unicode member paths

[Vhati]

BPO	17656
Nosy	@loewis, @birkenfeld, @terryjreedy, @gpshead, @amauryfa, @pitrou, @vstinner, @larryhastings, @tiran, @benjaminp, @ned-deily, @ezio-melotti, @bitdancer, @serhiy-storchaka, @koobs
Files	Kestrel Cruiser.zip zipfile_extract_unicode.patch test_extract_unicode_filenames_skip.patch: Skip test_extract_unicode_filenames

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/serhiy-storchaka'
closed_at = <Date 2013-05-13.00:06:27.162>
created_at = <Date 2013-04-08.03:20:59.521>
labels = ['type-bug', 'OS-windows', 'release-blocker']
title = 'Python 2.7.4 breaks ZipFile extraction of zip files with unicode member paths'
updated_at = <Date 2013-05-27.16:07:10.597>
user = 'https://bugs.python.org/Vhati'

bugs.python.org fields:

activity = <Date 2013-05-27.16:07:10.597>
actor = 'laci112'
assignee = 'serhiy.storchaka'
closed = True
closed_date = <Date 2013-05-13.00:06:27.162>
closer = 'benjamin.peterson'
components = ['Windows']
creation = <Date 2013-04-08.03:20:59.521>
creator = 'Vhati'
dependencies = []
files = ['29728', '29729', '29952']
hgrepos = []
issue_num = 17656
keywords = ['patch']
message_count = 24.0
messages = ['186264', '186265', '186273', '186275', '186285', '186326', '186444', '186490', '186493', '186494', '186496', '186660', '186703', '187430', '187453', '187461', '187474', '188583', '188730', '188731', '188767', '188769', '188779', '188780']
nosy_count = 22.0
nosy_names = ['loewis', 'georg.brandl', 'terry.reedy', 'gregory.p.smith', 'amaury.forgeotdarc', 'pitrou', 'vstinner', 'larry', 'christian.heimes', 'schmir', 'benjamin.peterson', 'ned.deily', 'ezio.melotti', 'Arfrever', 'r.david.murray', 'twb', 'neologix', 'catalin.iacob', 'python-dev', 'serhiy.storchaka', 'koobs', 'Vhati']
pr_nums = []
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue17656'
versions = ['Python 2.7']

[Vhati]

Python 2.7.4 fails while extracting zip files when 'member' is a unicode path.

---

Traceback (most recent call last):
  ...
    my_zip.extract(item, tmp_folder_path)
  File "D:\Apps\Python274\lib\zipfile.py", line 1024, in extract
    return self._extract_member(member, path, pwd)
  File "D:\Apps\Python274\lib\zipfile.py", line 1057, in _extract_member
    arcname = arcname.translate(table)
TypeError: character mapping must return integer, None or unicode

---

2.7.3 had no problems because the call to translate() is new.

The following, copied from ZipFile.py, will recreate the error.

import string
illegal = ':<>|"?*'
table = string.maketrans(illegal, '_' * len(illegal))

arcname = "hi"
arcname = arcname.translate(table)
# ascii strings are fine

arcname = u"hi"
arcname = arcname.translate(table)
# unicode fails
# Traceback (most recent call last):
#   File "<stdin>", line 1, in <module>
# TypeError: character mapping must return integer, None or unicode

---

I tried using unicode literals for the illegal string and maketrans underscore arg, but that didn't work. Suggestions?

Here's a link to the doc for translate().
http://docs.python.org/2/library/stdtypes.html#str.translate

[Vhati]

Apparently namelist() can return either ascii or unicode strings for its members, depending on the archive. Obviously this'd apply to literal unicode strings as well.

[Vhati]

Oops, passing a unicode literal to extract()'s member arg wouldn't be sufficient.

The extract() method quietly converts strings to ZipInfo objects via getinfo(member_string). Then _extract_member() takes the filename attribute of that ZipInfo object, which causes problems when when THAT is unicode.

So I guess this bug only applies to archives with unicode member paths.

Attached is one such file to aid in troubleshooting.

[loewis]

It appears that this is a consequence of the changes in bpo-6972, in particular change 4d1948689ee1.

[serhiy-storchaka]

Yes, it's my fault. Here is a patch (with test) which fixes this regression in 2.7. This is 2.7 only issue, in Python 3 arcnames always are unicode. Please test on Windows.

[Vhati]

The 2013-04-08 patch worked on Windows XP.

[pitrou]

Perhaps this would deserve a 2.7.5?

[benjaminp]

Yes; I won't have time for a few days, though.

[birkenfeld]

I guess I will join with 3.2 and 3.3 for bpo-17666.