SysLogHandler can't handle Unicode

[rszefler]

BPO	7077
Nosy	@birkenfeld, @vsajip, @vstinner
Files	utffixedsysloghandler.py: UTFFixedSysLogHandler, an overriden logging.handlers.SysLogHandlers that fixes BOM bug introduced in r75586

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/vsajip'
closed_at = <Date 2010-08-21.21:32:42.330>
created_at = <Date 2009-10-07.12:42:37.578>
labels = ['type-bug', 'library']
title = "SysLogHandler can't handle Unicode"
updated_at = <Date 2010-09-03.09:10:03.563>
user = 'https://bugs.python.org/rszefler'

bugs.python.org fields:

activity = <Date 2010-09-03.09:10:03.563>
actor = 'vinay.sajip'
assignee = 'vinay.sajip'
closed = True
closed_date = <Date 2010-08-21.21:32:42.330>
closer = 'georg.brandl'
components = ['Library (Lib)']
creation = <Date 2009-10-07.12:42:37.578>
creator = 'rszefler'
dependencies = []
files = ['18715']
hgrepos = []
issue_num = 7077
keywords = []
message_count = 14.0
messages = ['93694', '93724', '93734', '93737', '94141', '94323', '94937', '111646', '114413', '114422', '114566', '115382', '115402', '115424']
nosy_count = 5.0
nosy_names = ['georg.brandl', 'vinay.sajip', 'vstinner', 'rszefler', 'Remi.Broemeling']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'commit review'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue7077'
versions = ['Python 3.1', 'Python 2.7', 'Python 3.2']

[rszefler]

Trying to .emit() a Unicode string causes an awkward exception to be thrown:

Traceback (most recent call last):
  File "/usr/lib/python2.5/logging/handlers.py", line 672, in emit
    self.socket.sendto(msg, self.address)
TypeError: sendto() takes exactly 3 arguments (2 given)

The issue is fixed simply by adding some sort of encoding coercion
before the sendto, for example:

if type(msg)==unicode: msg=msg.encode('utf-8')

[vsajip]

To do this in a non-arbitrary way, it would make sense for the
SysLogHandler (and perhaps the other socket-based handlers, too) to grow
an optional encoding argument to their constructors, to be used to
encode when converting from unicode to str (str -> bytes for Py3K).

How does that sound?

[rszefler]

Fine with me, though problems would arise. Default encoding for example.
If encoding selection is mandatory it would break compatibility. Using
default locale is not such a good idea - local machine's locale would
generally not need to have any correlation to the remote logger's.

Maybe the best solution would be to coerce the text to ASCII per default
(such as not to break current semantics) but fix the exception thrown
(throw an Unicode*Error) and allow an optional encoding parameter to
handle non-ASCII characters?

[vsajip]

Robert Szefler added the comment:

Fine with me, though problems would arise. Default encoding for example.
If encoding selection is mandatory it would break compatibility. Using
default locale is not such a good idea - local machine's locale would
generally not need to have any correlation to the remote logger's.

I'm not planning to make encoding selection mandatory: I would provide a parameter encoding=None so backward compatibility is preserved.

On 2.x: During output, a check will be made for Unicode. If not found, the data is output as is. Otherwise (if Unicode) it's encoded using either the specified encoding (if not None) or some default - for example, locale.getpreferredencoding().

I understand what you're saying about the locales of two different machines not being the same - but there's no way around this, because if a socket receives bytes representing text, it needs to know what encoding was used so that it can reconstruct the Unicode correctly. So that information at least needs to be known at the receiving end, rather than guessd. While 'utf-8' might be a reasonable choice, I'm not sure it should be enforced. So the code sending the bytes can specify e.g. 'cp1251' and the other end has to know so it can decode correctly. I've posted on python-dev for advice about what encoding to use if none is specified.

On 3.x: We will always be passing Unicode in so we will always need to convert to bytes using some encoding. Again, if not specified, a suitable default encoding needs to be chosen.

Maybe the best solution would be to coerce the text to ASCII per default
(such as not to break current semantics) but fix the exception thrown
(throw an Unicode*Error) and allow an optional encoding parameter to
handle non-ASCII characters?

I'm not exactly sure what you mean, but I think I've covered it in my comments above. To summarise:

On 2.x, encoding is not mandatory but if Unicode is passed in, either a specified encoding or a suitable default encoding will be used to encode the Unicode into str.

On 3.x, encoding is not mandatory and Unicode should always be passed in, which will be encoded to bytes using either a specified encoding or a suitable default encoding.

[vsajip]

According to information from Martin von Löwis - see

http://mail.python.org/pipermail/python-dev/2009-October/092825.html

• UTF-8 should always be used, with a BOM, when sending Unicode
  (according to RFC 5424). The fix will use this approach. No encoding
  parameter will be added.

[vsajip]

Fix checked into trunk and py3k (r75586). Please verify in your
environment and post your results here.

There are no plans to backport this to 2.6 or earlier.

[vsajip]

No adverse feedback on fix, so closing.

[birkenfeld]

There is indeed a problem with the patch: the BOM is put in front of the angle brackets indicating the priority/facility, so the syslog can't find it anymore. The BOM should be put after the brackets.

[vsajip]

Updated implementation so that <n> + BOM + message is sent, for py3k branch only (r84218). Please verify fix in your environment.

[vsajip]

Err, make that r84222.