mimetypes initialization fails on Windows because of non-Latin characters in registry

[DmitryJemerov]

BPO	9291
Nosy	@loewis, @jaraco, @vstinner, @tjguk, @merwok, @bobince, @bitdancer, @briancurtin, @shimizukawa, @Fak3
Files	9291.patch 9291a.patch: Issue 9291 patch sitecustomize.py issue9291-key-utf8.ini: Offending REG key in Windows Registry file encoded with utf-8 issue9291-key.reg issue9291.8.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/tjguk'
closed_at = <Date 2014-04-29.17:27:43.543>
created_at = <Date 2010-07-18.11:54:53.841>
labels = ['easy', 'type-bug', 'library', 'OS-windows']
title = 'mimetypes initialization fails on Windows because of non-Latin characters in registry'
updated_at = <Date 2014-06-10.17:01:14.898>
user = 'https://bugs.python.org/DmitryJemerov'

bugs.python.org fields:

activity = <Date 2014-06-10.17:01:14.898>
actor = 'exarkun'
assignee = 'tim.golden'
closed = True
closed_date = <Date 2014-04-29.17:27:43.543>
closer = 'tim.golden'
components = ['Library (Lib)', 'Windows']
creation = <Date 2010-07-18.11:54:53.841>
creator = 'Dmitry.Jemerov'
dependencies = []
files = ['18143', '19332', '33268', '34187', '34188', '34983']
hgrepos = []
issue_num = 9291
keywords = ['patch', 'easy']
message_count = 32.0
messages = ['110637', '110760', '110881', '111288', '111291', '111318', '113662', '119362', '119364', '177044', '202755', '202840', '206494', '206510', '206528', '206540', '206579', '206938', '207076', '211921', '211931', '211932', '211934', '211935', '211936', '211938', '216571', '216903', '217137', '217271', '217273', '220175']
nosy_count = 24.0
nosy_names = ['loewis', 'exarkun', 'jaraco', 'vstinner', 'tim.golden', 'eric.araujo', 'kaizhu', 'aclover', 'r.david.murray', 'brian.curtin', 'Suzumizaki', 'frankoid', 'Dmitry.Jemerov', 'shimizukawa', 'vldmit', 'Vladimir Iofik', 'python-dev', 'Roman.Evstifeev', 'adamhj', 'me21', 'Hugo.Lol', 'Daniel.Szoska', 'Micha\xc5\x82.Pasternak', 'quick.es']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue9291'
versions = ['Python 2.7']

[DmitryJemerov]

On Windows, mimetypes initialization reads the list of MIME types from the Windows registry. It assumes that all characters are Latin-1 encoded, and fails when it's not the case, with the following exception:

Traceback (most recent call last):
  File "mttest.py", line 3, in <module>
    mimetypes.init()
  File "c:\Python27\lib\mimetypes.py", line 355, in init
    db.read_windows_registry()
  File "c:\Python27\lib\mimetypes.py", line 260, in read_windows_registry
    for ctype in enum_types(mimedb):
  File "c:\Python27\lib\mimetypes.py", line 250, in enum_types
    ctype = ctype.encode(default_encoding) # omit in 3.x!
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe0 in position 0: ordinal not in range(128)

This can be reproduced, for example, on a Russian Windows XP installation which has QuickTime installed (QuickTime creates the non-Latin entries in the registry). The following line causes the exception to happen:

import mimetypes; mimetypes.init()

[bitdancer]

I'm guessing this problem doesn't occur in 3.x? If so, the quick fix would be to have the registry code catch UnicodeError instead of UnicodeEncodeError. That may be the correct fix anyway.

The "fun" part of this bug is going to be creating a unit test for it.

[DmitryJemerov]

The problem doesn't happen on Python 3.1.2 because it doesn't have the code in mimetypes that accesses the Windows registry. Haven't tried the 3.2 alphas yet.

[DmitryJemerov]

Patch (suggested fix and unittest) attached.

[DmitryJemerov]

And by the way I've verified that the problem doesn't happen in py3k trunk.

[bitdancer]

And just for clarity: py3k trunk does contain the _winreg code path.

[kaizhu]

python 3.1.2 mimetypes initialization also fails in redhat linux:

>>> import http.server
Traceback (most recent call last):
  File "/home/public/i386-redhat-linux-gnu/python/lib/python3.1/http/server.py", line 588, in <module>
    class SimpleHTTPRequestHandler(BaseHTTPRequestHandler):
  File "/home/public/i386-redhat-linux-gnu/python/lib/python3.1/http/server.py", line 764, in SimpleHTTPRequestHandler
    mimetypes.init() # try to read system mime.types
  File "/home/public/i386-redhat-linux-gnu/python/lib/python3.1/mimetypes.py", line 305, in init
    db.readfp(open(file))
  File "/home/public/i386-redhat-linux-gnu/python/lib/python3.1/mimetypes.py", line 209, in readfp
    line = fp.readline()
  File "/home/public/i386-redhat-linux-gnu/bin/../python/lib/python3.1/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 3921: ordinal not in range(128)

[VladimirIofik]

Here is a better patch.

[VladimirIofik]

UnicodeDecodeException is thrown because 'ctype' is already a string,
so it is first implicitly decoded by default encoder (which is 'ascii') and then reencoded back. I see no reason in all these actions, so I simply removed them. I think Antoine Pitrou (who is the author of these lines) can shed some light on this, but I guess it's just a copy-paste of the code below.

[vstinner]

File "c:\Python27\lib\mimetypes.py", line 250, in enum_types
ctype = ctype.encode(default_encoding) # omit in 3.x!
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe0 in position 0: ordinal not in range(128)

The encoding is wrong. We should read the registry using Unicode, or at least use the correct encoding. The correct encoding is the ANSI code page: sys.getfilesystemencoding().

Can you please try with: default_encoding = sys.getfilesystemencoding() ?

python 3.1.2 mimetypes initialization also fails in redhat linux: (...)

In Python 3.3, MimeTypes.read() opens files in UTF-8. The issue bpo-13025 explains why UTF-8 is used instead the locale encoding, or another encoding.

I see that read_mime_types() uses the locale encoding, it looks like a bug, it should also use UTF-8.

[tjguk]

Only just been reminded of this one; it's possible that it's been superseded by bpo-15207. At the least, that issue resulted in a code change in this area of mimetypes. I'll have a look later.