Can not print utf-8(e.g. Chinese)

[kongp3]

What I wanted

• I want to print utf-8 like '哈哈哈' in my code

What I did

• I set configure in my main()

    structlog.configure(
        processors=[
            structlog.processors.UnicodeDecoder(),
            structlog.processors.KeyValueRenderer(
                key_order=['event', 'request_id'],
            ),
        ],
        context_class=structlog.threadlocal.wrap_dict(dict),
        logger_factory=structlog.stdlib.LoggerFactory(),
    )

• and print log

log = structlog.get_logger(request_id=shortuuid.uuid().upper())
log.info('哈哈哈', api='/api_card_save_kyc_data/')

What I get

• Print the code, not the text. The code u'\u54c8\u54c8\u54c8' below is '哈哈哈' in Chinese

event=u'\u54c8\u54c8\u54c8' request_id=u'HDRWLXJA9NWHQR3NWDSMLF' 

What I found

• I debugged the code I found it described below
• The magic method __call__ in structlog/processors.py the class KeyValueRenderer
• seems like “cook” the return value

        return ' '.join(k + '=' + repr(v)
                        for k, v in self._ordered_items(event_dict))

• The very repr make the v from '哈哈哈' to u'\u54c8\u54c8\u54c8'
• After I removed the repr I get the info below

event=哈哈哈 request_id=W3YHQZUNQDG4XDA2CTYYRE 

What the last

• I had tried many ways to solve this problem without changing the origin code, but failed.
• Maybe there is best way to deal it that I can't reach.
• If it is true, please kindly tell me ,and I will be appreciate it very much
• If it is false, could you make some change on the code above, to make it print utf-8 better
• thanks very much

@hynek

[hynek]

Is this Python 2.7? You need UnicodeEncoder instead of UnicodeDecoder on legacy Python versions.

[kongp3]

@hynek
Thanks for your commentary first ^_^
and then
I had tried both, and I had debugged it with the debug tool in Pycharm, and I tried it onece again when I saw your commentary. It was no helped. Here is my demo main code. Thank you for caring me again ^_^

import structlog
import tornado.options
from tornado import autoreload
from tornado.httpserver import HTTPServer
from tornado.ioloop import IOLoop
from tornado.wsgi import WSGIContainer
from flask import Flask

app = Flask(__name__)



if __name__ == '__main__':
    structlog.configure(
        processors=[
            structlog.processors.UnicodeEncoder(),
            structlog.processors.KeyValueRenderer(
                key_order=['event', 'request_id'],
            ),
        ],
        context_class=structlog.threadlocal.wrap_dict(dict),
        logger_factory=structlog.stdlib.LoggerFactory(),
    )
    tornado.options.parse_command_line()
    http_server = HTTPServer(WSGIContainer(app))
    http_server.listen(9998)
    instance = IOLoop.instance()
    autoreload.start(instance)
    instance.start()

[hynek]

OK I’ve looked further into this and it’s interesting!

Python 2:

>>> print(repr('哈哈哈'))
'\xe5\x93\x88\xe5\x93\x88\xe5\x93\x88'
>>> print(repr(u'哈哈哈'))
u'\u54c8\u54c8\u54c8'

Python 3:

>>> print(repr('哈哈哈'))
'哈哈哈'

Python 3 has simply a much better Unicode support.

I am not 100% sure what to make of it though. :(

[hynek]

So I think I found a workaround by adding a new option to (KeyValue|Console)Renderer:

>>> import structlog, sys, logging
>>> structlog.get_logger().info("哈哈哈", some_key="哈哈哈")
2016-12-09 12:52.52 哈哈哈                      some_key=哈哈哈
>>> handler = logging.StreamHandler(sys.stdout)
>>> root_logger = logging.getLogger()
>>> root_logger.addHandler(handler)
>>> structlog.configure(
...         processors=[
...             structlog.processors.UnicodeEncoder(),
...             structlog.processors.KeyValueRenderer(
...                 key_order=['event', 'request_id'], repr_native_str=False
...             ),
...         ],
...         context_class=structlog.threadlocal.wrap_dict(dict),
...         logger_factory=structlog.stdlib.LoggerFactory(),
...     )
>>> structlog.get_logger().warning("哈哈哈")
event=哈哈哈 request_id=None

It’s not on PyPI yet; would you mind testing it, if it works for you?

[kongp3]

OK, I'll test it, and will tell you the result ,tks~~

[kongp3]

It is worked!
And I am so exciting!!
I pip install form github, and the result is in the picture below
Thanks very much for solving this problem ~_~
If you push it to the PyPI, please kindly @ me here, thank you.

pip install git+https://github.com/hynek/structlog.git

[hynek]

It took a bit longer than I hoped, but 17.1 is on PyPI!

[kongp3]

OK, thanks.

[nathandouglas]

@ianbrayoni you can actually supply those kwargs to the JSONRenderer class through the dumps_kw argument. I ran into the same issue, but noticed this is the source code (snippet below):

class JSONRenderer(object):
    """
    Render the `event_dict` using ``serializer(event_dict, **json_kw)``.
    ....
    """
    def __init__(self, serializer=json.dumps, **dumps_kw):
        self._dumps_kw = dumps_kw
        self._dumps = serializer

    def __call__(self, logger, name, event_dict):
        return self._dumps(event_dict, default=_json_fallback_handler,
                           **self._dumps_kw)

[hynek]

I think you want structlog.processors.UnicodeEncoder.