New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

json extractor prefix #1646

Closed
runningman84 opened this Issue Dec 22, 2015 · 12 comments

Comments

Projects
None yet
9 participants
@runningman84

runningman84 commented Dec 22, 2015

Right now you can not add a prefix to the json parser, this means if a json contains fields like timestamp or message the root field is overwritten.

In this case the message cannot be written to elasticsearch

[0]: index [graylog2_0], type [message], id [20e39820-a8bd-11e5-8ec4-00163e2b2cfe], message [MapperParsingException[failed to parse [timestamp]]; nested: MapperParsingException[failed to parse date field [2015-12-22T15:03:10.353722+0000], tried both date format [yyyy-MM-dd HH:mm:ss.SSS], and timestamp number with locale []]; nested: IllegalArgumentException[Invalid format: "2015-12-22T15:03:10.353722+0000" is malformed at "T15:03:10.353722+0000"]; ]

This is the a sample field sensu:

{"timestamp":"2015-12-22T15:08:26.058363+0000","level":"warn","message":"reconnecting to transport"}

With prefix graylog could create new fields like
sensu_level
sensu_message
...

@runningman84 runningman84 changed the title from json extractor allow prefix to json extractor prefix Dec 22, 2015

@bernd bernd added the feature label Jan 29, 2016

@hai-ld

This comment has been minimized.

hai-ld commented Feb 23, 2016

+1

Log messages like this cannot be processed by Graylog, causing them to be dropped, which I think is a really bad and surprising behaviour. At the very least, this should be documented in JSON extractor page about reserved fields: what they are, how to overwrite them correctly, etc.

@alborq

This comment has been minimized.

alborq commented Mar 1, 2016

same problem with symfony exception JSON.... message field is override.

@benvon

This comment has been minimized.

benvon commented Mar 15, 2016

I am also seeing a very similar issue with the JSON extractor not applying ISO8601 timestamp format correctly, then dropping the message.

@jstop

This comment has been minimized.

jstop commented May 13, 2016

+1
Maybe this explains why my JSON extractor works on try it, but doesn't appear to actually work. My JSON has a message field.

@ulope

This comment has been minimized.

ulope commented Jun 17, 2016

This is a really big issue. Our messages all have a level field that contains a string level name (e.g. DEBUG, INFO, etc.)

Processing messages like that with the JSON extractor leads to this error: "[MapperParsingException[failed to parse [level]]; nested: NumberFormatException[For input string: "INFO"];]"

@levisbakalinsky

This comment has been minimized.

levisbakalinsky commented Jun 29, 2016

Are there any plans to get this feature added to the latest version of GrayLog2? Would love to be able to send JSON strings using FileBeat, and have GrayLog decode the messages.

@kroepke

This comment has been minimized.

Member

kroepke commented Jun 29, 2016

Is this only about adding a static prefix to the extractor configuration?

@levisbakalinsky

This comment has been minimized.

levisbakalinsky commented Jun 29, 2016

@kroepke. Yes, When the extracted JSON fields get added to the event, it would be ideal if it was something like prefix_message, prefix_timestamp, prefix_level, etc. Events with keys like (message, timestamp, level), get dropped by the JSON extractor.

@kroepke

This comment has been minimized.

Member

kroepke commented Jun 29, 2016

@levisbakalinsky Ok, that seems easy enough to do. I'll put it into 2.1, should be a quick fix.

@kroepke kroepke added this to the 2.1.0 milestone Jun 29, 2016

@ulope

This comment has been minimized.

ulope commented Jun 29, 2016

As a workaround I found you can add a "replace with regex" extractor before the json extractor and use that to rename the offending json key with ugly regex hackery...

@levisbakalinsky

This comment has been minimized.

levisbakalinsky commented Jun 29, 2016

@kroepke, thank you. Looking forward to that release.

@levisbakalinsky

This comment has been minimized.

levisbakalinsky commented Jun 29, 2016

@ulope, is it possible to reg/replace message & level in one go?
{"timeMillis":1467221355984,"thread":"http-exec-1","level":"WARN","loggerName":"company.logger.name","message":"","endOfBatch":false,"loggerFqcn":"org.apache.logging.log4j.spi.AbstractLogger"}

@bernd bernd self-assigned this Jul 14, 2016

bernd added a commit that referenced this issue Jul 14, 2016

edmundoa added a commit that referenced this issue Jul 15, 2016

Add whitespace replacement and prefix options to JSON extractor (#2481)
* Add JSON extractor option to replace whitespace in message keys

Fixes #2434

* Add JSON extractor option to add static prefix to every message key

Fixes #1646

* Disable whitespace replacement input when replacement is disabled

* Modify texts trying to add more context
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment