Add ECS log format

[m1keil]

Add ECS log format to curator, which is just JSON but with field names defined in the Elastic Common Schema.

Fixes #1519

[untergeek]

A little late for the question, but, have you verified that using dotted field notation works properly? Or does it map the JSON as field names with dots in them? It's only in master, so we can update this if it does map strangely. I've seen it work sometimes, and not others, so I want to be sure.

[untergeek]

Yeah. This won't work:

{"@timestamp": "2020-03-13T17:50:42.912Z", "log.level": "INFO", "log.logger": "curator.cli", "log.origin.file.line": 226, "log.origin.function": "run", "message": "Job completed."}

[m1keil]

Both ECS and ElasticSearch supports dot notation and unpack the fields into objects, but I can do nested dicts if you prefer? Do I miss anything regarding the dot notation?

[untergeek]

ECS and Elasticsearch support dot notation for queries.

The output I shared shows that the dotted notation is inside the JSON as a key name. This will definitely be inserted/parsed by Elasticsearch as a key name of {"foo.bar": "value"} and not {"foo": { "bar": "value"}}

Not to worry, I have already re-coded the original LogstashFormatter class to address this. I'll share the PR number here when I submit it in a few minutes.

[untergeek]

See #1531

[untergeek]

{"@timestamp": "2020-03-13T22:39:38.041Z", "log": {"level": "INFO", "logger": "curator.cli", "origin": {"file": {"line": 226}, "function": "run"}, "original": "Job completed."}, "message": "Job completed."}

I took the liberty of adding log.original as well.

[m1keil]

ECS and Elasticsearch support dot notation for queries.

Interesting. from my tests I assumed it just silently converts it to objects as described here.

However it seems like dot notation is still causing some grief so I guess outputting proper objects is easier.

Thanks for the fix!

[untergeek]

Yes. In 2.0, dotted notation was removed, then re-added in 2.4, then tweaked before being re-added for querying. Very confusing stuff. You see, if I'm querying something with dotted notation, I should eventually be able to find it. But if I'm trying to write with dotted notation, and one field in the path contains a period in the field name, then how is ES to know whether it should be nested or not?

So, for index writing, explicitly specifying nested paths is the best way to go.