Logging: Unicode Error

[guettli]

BPO	13232
Nosy	@vsajip, @vstinner, @ezio-melotti
Files	unicodedecodeerror-in-logging.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2011-10-24.22:26:40.149>
created_at = <Date 2011-10-20.14:43:19.447>
labels = ['type-bug', 'library']
title = 'Logging: Unicode Error'
updated_at = <Date 2011-10-24.22:26:40.148>
user = 'https://bugs.python.org/guettli'

bugs.python.org fields:

activity = <Date 2011-10-24.22:26:40.148>
actor = 'python-dev'
assignee = 'none'
closed = True
closed_date = <Date 2011-10-24.22:26:40.149>
closer = 'python-dev'
components = ['Library (Lib)']
creation = <Date 2011-10-20.14:43:19.447>
creator = 'guettli'
dependencies = []
files = ['23486']
hgrepos = []
issue_num = 13232
keywords = []
message_count = 4.0
messages = ['146018', '146019', '146066', '146332']
nosy_count = 5.0
nosy_names = ['vinay.sajip', 'guettli', 'vstinner', 'ezio.melotti', 'python-dev']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue13232'
versions = ['Python 2.7']

[guettli]

In changeset fe6be0426e0d the format() method was changed. Unfortunately it does not catch all unicode decode errors.

I think line 482 of logging/init.py should be modified:
to this (add 'replace'):

s = s + record.exc_text.decode(sys.getfilesystemencoding(), 'replace')

http://hg.python.org/cpython/file/f35514dfadf8/Lib/logging/__init__.py#l482

Here is the stacktrace we get:
{{{
Traceback (most recent call last):
  File "/usr/lib64/python2.7/logging/__init__.py", line 838, in emit
    msg = self.format(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 715, in format
    return fmt.format(record)
  File "/home/modbau_esg_p/djangotools/utils/logutils.py", line 32, in format
    msg=logging.Formatter.format(self, record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 482, in format
    s = s + record.exc_text.decode(sys.getfilesystemencoding())
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 662: ordinal not in range(128)
Logged from file base.py, line 209
}}}

[vsajip]

Can you tell me what the actual data was which failed to be decoded? Is there more than one encoding in effect (e.g. one for the filesystem, and another for the other data in the exception being logged)?

[guettli]

I attached a testcase (unicodedecodeerror-in-logging.py). If the filesystemencoding is UTF-8 and the source code is encoded in latin1, then the logging fails. It happens because there is a German umlaut in the comment behind 1/0.

I added 'replace' to the decode() in __init__.py and the it works. The German umlaut gets displayed as inverted question mark. But this is better than no logging message.

[python-dev]

New changeset 4bb1dc4e2cec by Vinay Sajip in branch '2.7':
Closes bpo-13232: Handle multiple encodings in exception logging.
http://hg.python.org/cpython/rev/4bb1dc4e2cec