logging.StreamHandler encodes log message in UTF-8

[vdvo]

BPO	835353
Nosy	@loewis, @vsajip

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/vsajip'
closed_at = <Date 2004-03-02.13:32:30.000>
created_at = <Date 2003-11-03.22:45:43.000>
labels = ['invalid', 'library']
title = 'logging.StreamHandler encodes log message in UTF-8'
updated_at = <Date 2004-03-02.13:32:30.000>
user = 'https://bugs.python.org/vdvo'

bugs.python.org fields:

activity = <Date 2004-03-02.13:32:30.000>
actor = 'vinay.sajip'
assignee = 'vinay.sajip'
closed = True
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2003-11-03.22:45:43.000>
creator = 'vdvo'
dependencies = []
files = []
hgrepos = []
issue_num = 835353
keywords = []
message_count = 5.0
messages = ['18886', '18887', '18888', '18889', '18890']
nosy_count = 3.0
nosy_names = ['loewis', 'vinay.sajip', 'vdvo']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue835353'
versions = ['Python 2.3']

[vdvo]

For some reason that I do not see,
logging.StreamHandler in Python 2.3 insists on writing
plain non-Unicode strings to the stream, and the
encoding is hard-coded as UTF-8:

            if not hasattr(types, "UnicodeType"): #if
no unicode support...
                self.stream.write("%s\n" % msg)
            else:
                try:
                    self.stream.write("%s\n" % msg)
                except UnicodeError:
                    self.stream.write("%s\n" %
msg.encode("UTF-8"))

This behaviour is neither documented nor reasonable.
Files can be perfectly able to write Unicode strings
(e.g., through the use of codecs.EncodedFile or with a
default encoding of sys.stdout), and even if they are
not, UTF-8 is hardly the only choice for an encoding. I
propose to simply replace the above code with:

self.stream.write(msg)
self.stream.write("\n")

[loewis]

Logged In: YES
user_id=21627

That would be an incompatible change, of course, as you then
may get encoding errors where you currently get none.

[vsajip]

Logged In: YES
user_id=308438

Notice that UTF-8 is only used if a UnicodeError is detected.
By default, "%s\n" % msg is written to the stream using the
stream's write(). If the stream can handle this without raising
a UnicodeError, then UTF-8 will not be used. Is there a
specific use case/test script which demonstrates a problem?

[vdvo]

Logged In: YES
user_id=545628

Hmmm... I can't remember what the exact problem was, but now
that I look at it again, I see that it must have been my
error. What a poor bug report this is. :-( Sorry.

Still, I'd like the encoding to be configurable: UTF-8 can
stay as the default, but it would be nice to have an option
to use, say, "iso-8859-2" or "windows-1250".

[vsajip]

Logged In: YES
user_id=308438

If you want to use some other encoding, why not use a
stream created using codecs.open(), and if necessary use a
Formatter which is Unicode-aware to convert from msg + args
to the formatted message? Then the exception handler should
never be invoked.

Or, do you mean, for the exception handler? I think UTF-8 is
OK as the default, since it is the most commonly used. I may
consider making this configurable for a future release, if there
is enough demand; for now you can patch it yourself.

I'll close this bug report now, I assume that's OK with you?