Logging to file does not accept UTF16

[pwronisz]

BPO	19267
Nosy	@vsajip, @jcea, @vstinner, @ezio-melotti, @hobbestigrou
Files	patch_utf16

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2013-10-15.21:48:18.573>
created_at = <Date 2013-10-15.15:08:24.144>
labels = ['type-bug', 'library', 'expert-unicode']
title = 'Logging to file does not accept UTF16'
updated_at = <Date 2013-10-16.02:31:55.634>
user = 'https://bugs.python.org/pwronisz'

bugs.python.org fields:

activity = <Date 2013-10-16.02:31:55.634>
actor = 'jcea'
assignee = 'none'
closed = True
closed_date = <Date 2013-10-15.21:48:18.573>
closer = 'python-dev'
components = ['Library (Lib)', 'Unicode']
creation = <Date 2013-10-15.15:08:24.144>
creator = 'pwronisz'
dependencies = []
files = ['32137']
hgrepos = []
issue_num = 19267
keywords = []
message_count = 10.0
messages = ['200001', '200002', '200003', '200013', '200014', '200019', '200022', '200025', '200026', '200027']
nosy_count = 7.0
nosy_names = ['vinay.sajip', 'jcea', 'vstinner', 'ezio.melotti', 'python-dev', 'pwronisz', 'hobbestigrou']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue19267'
versions = ['Python 2.7']

[pwronisz]

The following code reproduces the error:

import logging
logging.root.addHandler(logging.FileHandler(filename='test.log',encoding='UTF16'))
logging.error( u'b\u0142\u0105d')

I think the problem is in the line
logging/init.py:860: ufs = fs.decode(stream.encoding)

as Python can't really handle the following code:

fs = "%s\n"
print fs.decode('utf16') % u'foo'

[vstinner]

The example works fine on Python3:

localhost$ python3
Python 3.3.2 (default, Aug 23 2013, 19:00:04) 
[GCC 4.8.1 20130603 (Red Hat 4.8.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import logging
>>> logging.root.addHandler(logging.FileHandler(filename='test.log',encoding='UTF16'))
>>> logging.error( u'b\u0142\u0105d')
>>> 

localhost$ hexdump test.log
0000000 feff 0062 0142 0105 0064 000a
000000c

[vstinner]

I don't understand the purpose of fs.decode(stream.encoding): why not directly using ufs=u'%s\n"?

@pawel: Can you please try to replace ufs=fs.decode(stream.encoding) with ufs=u'%s\n'?

[pwronisz]

Hi Victor. Your fix works, but actually using simply ufs='%s\n' also seem to work, as
type( '%s' % u'foo')
and
type( u'%s' % u'foo')
returns the same - unicode.

So I would suggest dropping ufs completely, and changing the two occurences to fs. It works - I checked. Maybe some code refactoring could be done then, as I'm not sure if the two nested 'try' statements are needed in such case.

[pwronisz]

Btw I also don't see the purspose of fs.decode(stream.encoding), as it should rather be encoded and not decoded...

[hobbestigrou]

Hi,

here the patch to fix this bug on Python 2.7 with a test suite.

[vstinner]

patch_utf16 looks good to me.

[python-dev]

New changeset e94e29dab32c by Victor Stinner in branch '2.7':
Close bpo-19267: Fix support of multibyte encoding (ex: UTF-16) in the logging
http://hg.python.org/cpython/rev/e94e29dab32c

[vstinner]

This bug is specific to Python 2. I should now be fixed, thanks Paweł for the report and Natal for the patch.

@natal: Could you please sign the Contributor Agreement for next contributions?
http://www.python.org/psf/contrib/contrib-form/

[pwronisz]

Cool - looks good to me as well. Thanks guys.