extra fields #47
Comments
|
|
|
I have (had) only Fluent Handler and would wanted to see the extra fields on Fluent server. I guess this is a simple thought and wish. #log.debug('python fluent log test', extra={'extraField': 'test'} )
log.debug('python fluent log workaround. extraField: %s', 'test')despite it did the job, it is moving out of convention I guess. I had to convert all logging lines in the project to cover such. |
|
If you don't put the data in the format, the handler cannot know that you want it to output this to fluent. |
|
dkavraal , you can refer keys in extra in custom_format. For example, |
|
@yoichi The issue with the custom_format approach is that it converts all values to string. Consider the case where you'd want to add a numerical identifier: mylogger.exception("bad` error condition while processing job %d", job_id,
extra={'job_id': job_id})This log record would embed the job_id into the "message" field of the record (which necessarily is a string), but the Maybe there should be an option to copy all excess "extra" fields into the resulting dictionary that then gets serialized with msgpack and sent out to fluentd? |
|
@siretart why don't you pass structured data directly as msg (1st argument)? |
|
@yoichi because the logger might have other handlers attached, such as syslog or file hander, for which a dictionary would mess up existing logs. |
|
@yoichi I guess that could work. In the mean time, I came across https://github.com/vklochan/python-logstash/blob/master/logstash/formatter.py - which seems significantly simpler on the first look. I'll experiment with that a bit. |
|
@repeatedly in answer to your question from 2016, yes, other log handlers will accept the @yoichi's suggestion of logging a dict as msg works fine for Fluent Formatter, but it does not work with other log handlers that do not expect a structured message, and expect string message + It would be nice to use a single method of logging in the application that can be handled the same way by all handlers on the logger, including the standard Python ones. |
|
@adeadman I'm confused. You want to use I think that FluentHandler can handle other formatters. Setting up alternative logging schematics is not that difficult. This works in production perfectly: if self.fluent_url:
h = self.get_fluent_handler()
filter = self.get_fluent_overwriting_filter()
h.addFilter(filter)
f = self.get_fluent_formatter()
else:
h = StreamHandler()
f = Formatter("%(asctime)s %(levelname)s "
"%(name)s: %(message)s")
f.converter = time.gmtime
h.setFormatter(f)
root_logger.addHandler(h)
root_logger.setLevel(WARNING)
self.logger.setLevel(self.loglevel)
self.logger = LoggerAdapter(self.logger, key_map=DEFAULT_LOG_KEY_MAP) |
|
@arcivanov I am using However, the If the At the moment I'm working on adding support for extracting the |
|
Sentry is having the same problem as described here. A solution extracting the extra data can be found here |
|
Per Python documentation:
Inserting general-purpose logic into special circumstances feature is, generally, ill-advised. I'll close this as undesirable. |
|
@arcivanov First of all, thanks for a quick and constructive feedback. I'll take my right to disagree though and hope you'll reconsider your decision :) My main point is that so far no one has presented a clear use case for such feature of this library. Allow me to do it. Real life use-case
This works out of the box with other commonly used logging libraries (like Sentry).
Please consider that all major and popular Python logging libraries handle that use-case out of the box. Whereas fluent-logger-python takes by surprise by not supporting it and requiring the user to read through issues to find clues about how to make it work. It may seriously hinder the adoption.
Actually, I found it in every library I thought about. Referring to you concerns:
Personally, I see three solutions:
|
|
Thanks for your feedback. I still disagree. Let's consider two scenarios:
Right now people who need extra field functionality, as it's designed to be used in Python i.e. to augment log entries with data that is mostly not directly specified during logging call, have their own current methodologies for dealing with the extra fields. Some may fold them directly into the record fields, some may want them in extra field (like you), some may want to use them, some may derive other values from them etc. Let's look at the legislative intent, so to speak, of the Python logger in general and FluentD logger in particular. In Python logger extra fields are flat, folded in as fields of the In FluentD logger you can log a dictionary, message or JSON encoded in string, you can specify a dictionary of fields to log, you can switch to In fact this sample code works perfectly fine logging extra fields happily, preserving both types and structure: from logging import (root as root_logger,
Handler)
from pprint import pprint
from fluent.handler import FluentRecordFormatter
class TestHandler(Handler):
def emit(self, record):
pprint(self.format(record))
def main():
f = FluentRecordFormatter(exclude_attrs=())
h = TestHandler()
h.setFormatter(f)
root_logger.addHandler(h)
EXTRA = {"x": 1, "y": 0.1, "z": [1, "array", "of", b"basic", "types"]}
root_logger.error({"Hello": "Dict World with no extra :("})
root_logger.error({"Hello": "Dict World with extra! :)"},
extra=EXTRA)
root_logger.error("Hello World with no extra :(")
root_logger.error("Hello World with extra! :)",
extra=EXTRA)producing the following output (less the snippage): Now, you propose to add tons of specialized handling into fluent-logger that:
If we were to add this feature, every user that ever used extra before according to their needs would now find an You have to configure a |
I wasn't able to pass extra fields. Have you tested such:
on the last line I am expecting fluentd(td-agent) to pass the extraField into the collector.
Is there sth I'm missing?
The text was updated successfully, but these errors were encountered: