Elasticsearch output - dots in key names

[cert-lv]

Hi,

Modern Elasticsearch supports dots in key names. Currently an output bot does a replacement ("." to "_" by default) without any options.

One solution could be to add a bool config switch to enable/disable the replacement. (Somewhen in the future, probably, this func could be removed at all).

Fast solution is to replace dots with dots.

Also this ticked is opened as you wanted to change the default/name of the parameter.

[cert-lv]

extra keys are also concatenated with "_" without choice.
Result extra.somekey is allowed now, and is kind of more JSON-ish.

[navtej]

@cert-lv last i checked ES stopped allowing "." in key name. You can use the replacement_char option, set its value to '.' and you should be good to go, if your ES instance supports '.''s. IIRC after 1.X they dropped support for dots. Even before that the usage of dots was discouraged. However I haven't been following ES for a while now. May be that changed. Could you please let us know your ES version?

[cert-lv]

Hello, @navtej ,

Yes, I remember the period when dots were not allowed. Now we use ES 6.x (with 5.x it was
Ok too), but it seems dots were fixed a long time ago: https://www.elastic.co/guide/en/elasticsearch/reference/2.4/dots-in-names.html

In short - Elasticsearch 2.x does not support dots-to-object transformation and so dots in field names are not allowed in versions 2.0 - 2.3. In modern ES such fields are converted into multilevel objects.

[navtej]

@cert-lv does adding dot as replacement_char works for you?

[cert-lv]

@navtej, yes, as I've mentioned fast solution is to replace dots with dots, it works.

I've also noticed, that putting nothing in flatten_fields disables concatenation of extra fields with "_", so the second question is solved too.

But one more question - currently you put everything into one single ES indice, intelmq by default. May I send a Pull Request with a small fix to split indices into names like intelmq-MM-YYYY ? So users can manage/delete/backup indices by month.

[navtej]

@cert-lv please see this, https://github.com/navtej/intelmq/blob/latest/intelmq/bots/outputs/elasticsearch/output.py . it uses dynamic index based on date and time from time observation. I havent been able to make it generic because people may have vastly varied requirements.

[navtej]

I guess we can close this issue or we still have something open on it?

[ghost]

IMHO what has been discussed/explained here should go to the bot's documentation.

And I suggest that the replacement char parameter could be made optional (handle nonexistence and empy) if it is not necessary anymore.

[ghost]

Anyone willing to update the documentation with the findings mentioned here?

[navtej]

@cert-lv do you want to take it up.

[cert-lv]

https://github.com/certtools/intelmq/blob/master/docs/Bots.md#elasticsearch-output-bot - seems to already contain the info regarding this issue, so I'll close it.