Second proposal for JSON support #1143
I tried another option for #1069. The main change is that JSON processing now happens before multiline, so the order is:
The main advantage of this over #1069 is that it supports uses cases like Docker where normal log lines are wrapped in JSON. It should also work fine for most of the structured logging use cases.
Here is a sample config:
The idea is that when configuring the JSON decoder, you can select a "message" key that will be used in the next stages (multiline and line filtering). If you don't choose a "message" key but still try to configure line filtering or multiline, you will get a configuration error.
Compared to the #1069, this is more complex and contains a bit more corner cases (e.g. what happens if the text key is not a string) but the code is still simple enough I think.
This still requires the JSON objects to be one per line, but I think that's the safer assumption to make anyway (see comment from #1069).
Yes, it's more powerful and not a lot more complex. For sure even more powerful options can be imagined, but those would move us to much in the direction of "generic processing". Then, if I don't hear any objections, I'll move ahead to add tests and docs to this PR.
There seems to be an error in the OS build: https://travis-ci.org/elastic/beats/jobs/116909985#L1527
LGTM. I added some late thought about the config naming (sorry for not brining that up earlier), but we can move this also to a later stage. Please also update the CHANGELOG file.
Should we add a flag to the event when it was json decoded? Similar to what was requested for multiline?
referenced this pull request
Mar 22, 2016
Are there any proposals for multiline json support?
I see in #1069 there are some comments about it.
IMO a new
I think one of the primary use cases for logs are that they are human readable. The first thing I usually do when an issue arrises is to open up a console and scroll through the log(s). Filebeats provides
Using pretty printed JSON objects as log "lines" is nice because they are human readable.
Limiting the input to single line JSON objects limits the human usefulness of the log.
For example, here is a real-ish log line that I just grabbed:
The pretty printed JSON is much more human readable than the single line format :)
I understand it might be out of scope for this pull request, but I'm hoping filebeats can eventually support it.