New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can not print utf-8(e.g. Chinese) #94

Closed
kongp3 opened this Issue Dec 1, 2016 · 10 comments

Comments

Projects
None yet
3 participants
@kongp3

kongp3 commented Dec 1, 2016

What I wanted

  • I want to print utf-8 like '哈哈哈' in my code

What I did

  • I set configure in my main()
    structlog.configure(
        processors=[
            structlog.processors.UnicodeDecoder(),
            structlog.processors.KeyValueRenderer(
                key_order=['event', 'request_id'],
            ),
        ],
        context_class=structlog.threadlocal.wrap_dict(dict),
        logger_factory=structlog.stdlib.LoggerFactory(),
    )
  • and print log
log = structlog.get_logger(request_id=shortuuid.uuid().upper())
log.info('哈哈哈', api='/api_card_save_kyc_data/')

What I get

  • Print the code, not the text. The code u'\u54c8\u54c8\u54c8' below is '哈哈哈' in Chinese
event=u'\u54c8\u54c8\u54c8' request_id=u'HDRWLXJA9NWHQR3NWDSMLF' 

What I found

  • I debugged the code I found it described below
  • The magic method __call__ in structlog/processors.py the class KeyValueRenderer
  • seems like “cook” the return value
        return ' '.join(k + '=' + repr(v)
                        for k, v in self._ordered_items(event_dict))
  • The very repr make the v from '哈哈哈' to u'\u54c8\u54c8\u54c8'
  • After I removed the repr I get the info below
event=哈哈哈 request_id=W3YHQZUNQDG4XDA2CTYYRE 

What the last

  • I had tried many ways to solve this problem without changing the origin code, but failed.
  • Maybe there is best way to deal it that I can't reach.
  • If it is true, please kindly tell me ,and I will be appreciate it very much
  • If it is false, could you make some change on the code above, to make it print utf-8 better
  • thanks very much

@hynek

@hynek

This comment has been minimized.

Owner

hynek commented Dec 1, 2016

Is this Python 2.7? You need UnicodeEncoder instead of UnicodeDecoder on legacy Python versions.

@kongp3

This comment has been minimized.

kongp3 commented Dec 2, 2016

@hynek
Thanks for your commentary first ^_^
and then
I had tried both, and I had debugged it with the debug tool in Pycharm, and I tried it onece again when I saw your commentary. It was no helped. Here is my demo main code. Thank you for caring me again ^_^

import structlog
import tornado.options
from tornado import autoreload
from tornado.httpserver import HTTPServer
from tornado.ioloop import IOLoop
from tornado.wsgi import WSGIContainer
from flask import Flask

app = Flask(__name__)



if __name__ == '__main__':
    structlog.configure(
        processors=[
            structlog.processors.UnicodeEncoder(),
            structlog.processors.KeyValueRenderer(
                key_order=['event', 'request_id'],
            ),
        ],
        context_class=structlog.threadlocal.wrap_dict(dict),
        logger_factory=structlog.stdlib.LoggerFactory(),
    )
    tornado.options.parse_command_line()
    http_server = HTTPServer(WSGIContainer(app))
    http_server.listen(9998)
    instance = IOLoop.instance()
    autoreload.start(instance)
    instance.start()
@hynek

This comment has been minimized.

Owner

hynek commented Dec 9, 2016

OK I’ve looked further into this and it’s interesting!

Python 2:

>>> print(repr('哈哈哈'))
'\xe5\x93\x88\xe5\x93\x88\xe5\x93\x88'
>>> print(repr(u'哈哈哈'))
u'\u54c8\u54c8\u54c8'

Python 3:

>>> print(repr('哈哈哈'))
'哈哈哈'

Python 3 has simply a much better Unicode support.

I am not 100% sure what to make of it though. :(

hynek added a commit that referenced this issue Dec 9, 2016

@hynek

This comment has been minimized.

Owner

hynek commented Dec 9, 2016

So I think I found a workaround by adding a new option to (KeyValue|Console)Renderer:

>>> import structlog, sys, logging
>>> structlog.get_logger().info("哈哈哈", some_key="哈哈哈")
2016-12-09 12:52.52 哈哈哈                      some_key=哈哈哈
>>> handler = logging.StreamHandler(sys.stdout)
>>> root_logger = logging.getLogger()
>>> root_logger.addHandler(handler)
>>> structlog.configure(
...         processors=[
...             structlog.processors.UnicodeEncoder(),
...             structlog.processors.KeyValueRenderer(
...                 key_order=['event', 'request_id'], repr_native_str=False
...             ),
...         ],
...         context_class=structlog.threadlocal.wrap_dict(dict),
...         logger_factory=structlog.stdlib.LoggerFactory(),
...     )
>>> structlog.get_logger().warning("哈哈哈")
event=哈哈哈 request_id=None

It’s not on PyPI yet; would you mind testing it, if it works for you?

@kongp3

This comment has been minimized.

kongp3 commented Dec 11, 2016

OK, I'll test it, and will tell you the result ,tks~~

@kongp3

This comment has been minimized.

kongp3 commented Dec 11, 2016

It is worked!
And I am so exciting!!
I pip install form github, and the result is in the picture below
Thanks very much for solving this problem ~_~
If you push it to the PyPI, please kindly @ me here, thank you.

pip install git+https://github.com/hynek/structlog.git
image

@kongp3 kongp3 closed this Feb 7, 2017

@hynek

This comment has been minimized.

Owner

hynek commented Apr 24, 2017

It took a bit longer than I hoped, but 17.1 is on PyPI!

@kongp3

This comment has been minimized.

kongp3 commented Apr 28, 2017

OK, thanks.

@nathandouglas

This comment has been minimized.

nathandouglas commented Mar 9, 2018

@ianbrayoni you can actually supply those kwargs to the JSONRenderer class through the dumps_kw argument. I ran into the same issue, but noticed this is the source code (snippet below):

class JSONRenderer(object):
    """
    Render the `event_dict` using ``serializer(event_dict, **json_kw)``.
    ....
    """
    def __init__(self, serializer=json.dumps, **dumps_kw):
        self._dumps_kw = dumps_kw
        self._dumps = serializer

    def __call__(self, logger, name, event_dict):
        return self._dumps(event_dict, default=_json_fallback_handler,
                           **self._dumps_kw)
@hynek

This comment has been minimized.

Owner

hynek commented Mar 13, 2018

I think you want structlog.processors.UnicodeEncoder.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment