json extractor prefix

[runningman84]

Right now you can not add a prefix to the json parser, this means if a json contains fields like timestamp or message the root field is overwritten.

In this case the message cannot be written to elasticsearch

[0]: index [graylog2_0], type [message], id [20e39820-a8bd-11e5-8ec4-00163e2b2cfe], message [MapperParsingException[failed to parse [timestamp]]; nested: MapperParsingException[failed to parse date field [2015-12-22T15:03:10.353722+0000], tried both date format [yyyy-MM-dd HH:mm:ss.SSS], and timestamp number with locale []]; nested: IllegalArgumentException[Invalid format: "2015-12-22T15:03:10.353722+0000" is malformed at "T15:03:10.353722+0000"]; ]

This is the a sample field sensu:

{"timestamp":"2015-12-22T15:08:26.058363+0000","level":"warn","message":"reconnecting to transport"}

With prefix graylog could create new fields like
sensu_level
sensu_message
...

[hai-ld]

+1

Log messages like this cannot be processed by Graylog, causing them to be dropped, which I think is a really bad and surprising behaviour. At the very least, this should be documented in JSON extractor page about reserved fields: what they are, how to overwrite them correctly, etc.

[alborq]

same problem with symfony exception JSON.... message field is override.

[benvon]

I am also seeing a very similar issue with the JSON extractor not applying ISO8601 timestamp format correctly, then dropping the message.

[jstop]

+1
Maybe this explains why my JSON extractor works on try it, but doesn't appear to actually work. My JSON has a message field.

[ulope]

This is a really big issue. Our messages all have a level field that contains a string level name (e.g. DEBUG, INFO, etc.)

Processing messages like that with the JSON extractor leads to this error: "[MapperParsingException[failed to parse [level]]; nested: NumberFormatException[For input string: "INFO"];]"

[levisbakalinsky]

Are there any plans to get this feature added to the latest version of GrayLog2? Would love to be able to send JSON strings using FileBeat, and have GrayLog decode the messages.

[kroepke]

Is this only about adding a static prefix to the extractor configuration?

[levisbakalinsky]

@kroepke. Yes, When the extracted JSON fields get added to the event, it would be ideal if it was something like prefix_message, prefix_timestamp, prefix_level, etc. Events with keys like (message, timestamp, level), get dropped by the JSON extractor.

[kroepke]

@levisbakalinsky Ok, that seems easy enough to do. I'll put it into 2.1, should be a quick fix.

[ulope]

As a workaround I found you can add a "replace with regex" extractor before the json extractor and use that to rename the offending json key with ugly regex hackery...

[levisbakalinsky]

@kroepke, thank you. Looking forward to that release.

[levisbakalinsky]

@ulope, is it possible to reg/replace message & level in one go?
{"timeMillis":1467221355984,"thread":"http-exec-1","level":"WARN","loggerName":"company.logger.name","message":"","endOfBatch":false,"loggerFqcn":"org.apache.logging.log4j.spi.AbstractLogger"}