New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add LogRecordCompatibleDictRenderer processor #98

Closed
wants to merge 6 commits into
base: master
from

Conversation

Projects
None yet
3 participants
@sky-code
Contributor

sky-code commented Jan 10, 2017

Add new processor for easy integration with stdlib
basically LogRecordCompatibleDictRenderer forward structlog event_dict to stdlib logger

@codecov-io

This comment has been minimized.

codecov-io commented Jan 10, 2017

Current coverage is 100% (diff: 100%)

Merging #98 into master will not change coverage

@@           master   #98   diff @@
===================================
  Files          13    13          
  Lines         753   766    +13   
  Methods         0     0          
  Messages        0     0          
  Branches       95    96     +1   
===================================
+ Hits          753   766    +13   
  Misses          0     0          
  Partials        0     0          

Powered by Codecov. Last update 4db913e...2ae3274

sky-code added some commits Jan 10, 2017

@hynek

This comment has been minimized.

Owner

hynek commented Jan 11, 2017

This sounds interesting! Is it basically an inverted adapter to funnel structlog into logging and format it there freely? So you could use for example https://github.com/madzak/python-json-logger and have json everywhere? Is it something you’re already using yourself?

@sky-code

This comment has been minimized.

Contributor

sky-code commented Jan 11, 2017

Yes it is basically render event_dict into logging than you can handle it by using any logging handler and formatter. I use it already in my project, here is config:

structlog.configure(
        processors=[
            structlog.stdlib.PositionalArgumentsFormatter(),
            structlog.processors.StackInfoRenderer(),
            structlog.processors.format_exc_info,
            LogRecordDictRenderer(True, 'extra')
        ],
        context_class=dict,
        logger_factory=structlog.stdlib.LoggerFactory(),
        wrapper_class=structlog.stdlib.BoundLogger,
        cache_logger_on_first_use=True,
    )

and than config for stdlib logging

disable_existing_loggers = false
version = 1

[formatters.colored]
style = '$'
class = 'colorlog.ColoredFormatter'
format = '${log_color}${asctime} - ${name} - ${levelname} - ${message}'

[formatters.colored_extra]
style = '$'
class = 'colorlog.ColoredFormatter'
format = '${log_color}${asctime} - ${name} - ${levelname} - ${message} - ${extra}'


[handlers.console]
class = "logging.StreamHandler"
formatter = "colored"
level = "DEBUG"
stream = "ext://sys.stdout"

[handlers.console_extra]
class = "logging.StreamHandler"
formatter = "colored_extra"
level = "DEBUG"
stream = "ext://sys.stdout"

[handlers.logstash]
level = 'INFO'
class = 'logstash.TCPLogstashHandler'
host = 'localhost'
port = 9100 # Default value: 5959
version = 1 # Version of logstash event schema. Default value: 0 (for backward compatibility of the library)
message_type = 'logstash' # 'type' field in logstash message. Default value: 'logstash'.
fqdn = false # Fully qualified domain name. Default value: false.
# tags = ['tag1', 'tag2'] # list of tags. Default: None.

[loggers.natrix]
handlers = ["console_extra", "logstash"]
level = "DEBUG"
propagate = false

[root]
handlers = ["console", "logstash"]
level = "INFO"

Here you can see 2 formatter, one for root logger and one extended to display extra field which contains all data from event_dict, this give same result as using KeyValueRenderer, just for development and debug you will see all what you have. Than I use handler from python-logstash package to send logs into logstash with all data, but this logstash logger handler filter out extra field which duplicate data at general for debugging and not needed in logs storage
In general with this config you can use any stdlib handler and formatter to do what you want and at same time using features from structlog

@sky-code

This comment has been minimized.

Contributor

sky-code commented Jan 11, 2017

I was confused that structlog not have builtin way to forward log message with data what you have to stdlib logging for handling it here by using all handlers available already.
In my opinion structlog should have some builtin processor for integration with stdlib logging

If you accept this pull, consider next things

  • move LogRecordCompatibleDictRenderer to stdlib module or leave it in processors module?
  • rename LogRecordCompatibleDictRenderer to some more human friendly and pythonic
  • rename add_extra_event_dict argument
  • renmae add_event_dict_with_key argument
@hynek

This comment has been minimized.

Owner

hynek commented Jan 11, 2017

The lacking stdlib supporting is simply due to the fact that I try to avoid to use it myself as well as I can (it’s painfully slow and complex) and nobody stepped up so far to tackle the harder problems. :)

@hynek hynek referenced this pull request Mar 5, 2017

Closed

Please cut a new release #104

@hynek hynek closed this in efae84e Mar 6, 2017

@hynek

This comment has been minimized.

Owner

hynek commented Mar 6, 2017

First, please accept my apology, that it took so long for me to take care of this.

As you may see in the final commit, I have changed quite a bit especially because this function is a logging hotspot and should be as fast as possible. I believe the options you had are better handled with additional processors when you configure structlog for development. Of course, I’ve let you still be the author.

Many thanks though for this simply yet genius idea. I have promoted your approach as the main approach in the docs because that’s what many people asked for. This should have been in structlog from day 1!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment