Feature to enable/disable nested json parsing

[suraj-soni]

Add a feature-flag that allows to that replaces a field containing json as a string with a field that holds that json as valid json object.

[elasticsearch-release]

Jenkins standing by to test this. If you aren't a maintainer, you can ignore this comment. Someone with commit access, please review this and clear it for Jenkins to run.

[elasticsearch-release]

Jenkins standing by to test this. If you aren't a maintainer, you can ignore this comment. Someone with commit access, please review this and clear it for Jenkins to run.

[ruflin]

@suraj-soni Thanks a lot for the addition. I assume this is mainly for some docker logs?

We currently have quite a few discussions on how much processing we should add on the filebeat side, means should be done on the edge machine and which parts we prefer to see on Logstash or elasticsearch ingest. Every added feature on the client adds complexity and code we have to maintain. In addition filebeat gets more and more resource hungry. Obviously some of the features are also very nice to have on the client as this PR allows to apply more filtering of the json document directly in filebeat to reduce the amount of data sent. We will discuss this in the team and get back to you.

[tsg]

Hi @suraj-soni, thanks for the PR. We'd like to better understand your goals here, because in general we're trying to avoid features that do processing on the Beats side. In this particular case, doing this in Logstash or (soon) inside Elasticsearch itself (the Elasticsearch Ingest Node is getting a JSON processor) might be more convenient. Would the new JSON process inside Ingest Node fill your needs?

If we are to add this functionality to Beats, we'd prefer to do it as a processor, which will give it more flexibility in configuration.

[suraj-soni]

Hi @tsg, @ruflin,

Thanks for your inputs!

I see this feature architecturally as something that allows a consumer to be independent of the output that filebeat writes to. For ex. imagine a use-case to write directly to Kafka or Redis. So if the consumer is reading from ES (parsed JSON) or Kafka, the way to parse the data still remains the same.

If we are to add this functionality to Beats, we'd prefer to do it as a processor, which will give it more flexibility in configuration.

I like the idea of implementing this with a processor like 'parse_field' or similar so that you are not doing a parse on all the fields and only the ones that are specifically required. This would of course require a syntax definition for defining which fields should be parsed.

[tsg]

@suraj-soni Sorry for the delay here. We've discussed it internally, and we think that this functionality would fit Filebeat pretty well, if done in the form of a processor. The configuration could be something like:

processors:
  - json_decode.fields: ["message"]

For examples of other processors, see around here.

So if you are still interested, we'd be happy to work with you do get this change in.

I am closing this PR for now, as I assume you'd rather open another one for the new implementation.

[suraj-soni]

@tsg : Thanks for getting back. Yes i can still work on integrating the same feature but as a processor. Shall create a pull request soon.