Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add LogRecordCompatibleDictRenderer processor #98

Closed
wants to merge 6 commits into from
Closed

Add LogRecordCompatibleDictRenderer processor #98

wants to merge 6 commits into from

Conversation

@sky-code
Copy link
Contributor

@sky-code sky-code commented Jan 10, 2017

Add new processor for easy integration with stdlib
basically LogRecordCompatibleDictRenderer forward structlog event_dict to stdlib logger

@codecov-io
Copy link

@codecov-io codecov-io commented Jan 10, 2017

Current coverage is 100% (diff: 100%)

Merging #98 into master will not change coverage

@@           master   #98   diff @@
===================================
  Files          13    13          
  Lines         753   766    +13   
  Methods         0     0          
  Messages        0     0          
  Branches       95    96     +1   
===================================
+ Hits          753   766    +13   
  Misses          0     0          
  Partials        0     0          

Powered by Codecov. Last update 4db913e...2ae3274

@hynek
Copy link
Owner

@hynek hynek commented Jan 11, 2017

This sounds interesting! Is it basically an inverted adapter to funnel structlog into logging and format it there freely? So you could use for example https://github.com/madzak/python-json-logger and have json everywhere? Is it something you’re already using yourself?

@sky-code
Copy link
Contributor Author

@sky-code sky-code commented Jan 11, 2017

Yes it is basically render event_dict into logging than you can handle it by using any logging handler and formatter. I use it already in my project, here is config:

structlog.configure(
        processors=[
            structlog.stdlib.PositionalArgumentsFormatter(),
            structlog.processors.StackInfoRenderer(),
            structlog.processors.format_exc_info,
            LogRecordDictRenderer(True, 'extra')
        ],
        context_class=dict,
        logger_factory=structlog.stdlib.LoggerFactory(),
        wrapper_class=structlog.stdlib.BoundLogger,
        cache_logger_on_first_use=True,
    )

and than config for stdlib logging

disable_existing_loggers = false
version = 1

[formatters.colored]
style = '$'
class = 'colorlog.ColoredFormatter'
format = '${log_color}${asctime} - ${name} - ${levelname} - ${message}'

[formatters.colored_extra]
style = '$'
class = 'colorlog.ColoredFormatter'
format = '${log_color}${asctime} - ${name} - ${levelname} - ${message} - ${extra}'


[handlers.console]
class = "logging.StreamHandler"
formatter = "colored"
level = "DEBUG"
stream = "ext://sys.stdout"

[handlers.console_extra]
class = "logging.StreamHandler"
formatter = "colored_extra"
level = "DEBUG"
stream = "ext://sys.stdout"

[handlers.logstash]
level = 'INFO'
class = 'logstash.TCPLogstashHandler'
host = 'localhost'
port = 9100 # Default value: 5959
version = 1 # Version of logstash event schema. Default value: 0 (for backward compatibility of the library)
message_type = 'logstash' # 'type' field in logstash message. Default value: 'logstash'.
fqdn = false # Fully qualified domain name. Default value: false.
# tags = ['tag1', 'tag2'] # list of tags. Default: None.

[loggers.natrix]
handlers = ["console_extra", "logstash"]
level = "DEBUG"
propagate = false

[root]
handlers = ["console", "logstash"]
level = "INFO"

Here you can see 2 formatter, one for root logger and one extended to display extra field which contains all data from event_dict, this give same result as using KeyValueRenderer, just for development and debug you will see all what you have. Than I use handler from python-logstash package to send logs into logstash with all data, but this logstash logger handler filter out extra field which duplicate data at general for debugging and not needed in logs storage
In general with this config you can use any stdlib handler and formatter to do what you want and at same time using features from structlog

@sky-code
Copy link
Contributor Author

@sky-code sky-code commented Jan 11, 2017

I was confused that structlog not have builtin way to forward log message with data what you have to stdlib logging for handling it here by using all handlers available already.
In my opinion structlog should have some builtin processor for integration with stdlib logging

If you accept this pull, consider next things

  • move LogRecordCompatibleDictRenderer to stdlib module or leave it in processors module?
  • rename LogRecordCompatibleDictRenderer to some more human friendly and pythonic
  • rename add_extra_event_dict argument
  • renmae add_event_dict_with_key argument
@hynek
Copy link
Owner

@hynek hynek commented Jan 11, 2017

The lacking stdlib supporting is simply due to the fact that I try to avoid to use it myself as well as I can (it’s painfully slow and complex) and nobody stepped up so far to tackle the harder problems. :)

@hynek hynek mentioned this pull request Mar 5, 2017
@hynek hynek closed this in efae84e Mar 6, 2017
@hynek
Copy link
Owner

@hynek hynek commented Mar 6, 2017

First, please accept my apology, that it took so long for me to take care of this.

As you may see in the final commit, I have changed quite a bit especially because this function is a logging hotspot and should be as fast as possible. I believe the options you had are better handled with additional processors when you configure structlog for development. Of course, I’ve let you still be the author.

Many thanks though for this simply yet genius idea. I have promoted your approach as the main approach in the docs because that’s what many people asked for. This should have been in structlog from day 1!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

3 participants