Dedot kubernetes fields

[walktall]

Related to #5942

@ruflin @exekias PTAL~

[elasticmachine]

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

[ruflin]

Thanks for adding this option here. Could you add a test case for this "switch" option?

[walktall]

@ruflin Tests added, and also made some changes to original test code...

[ruflin]

@exekias This is a breaking change. I wonder if you see an issue in this?

[exekias]

We need this as a bugfix anyway, isn't it? Didn't follow previous conversations but I guess this is needed to avoid mapping issues?

In any case it's a breaking change and we should make it clear in the release notes

[ruflin]

As far as I know it didn't break yet but it would pretty soon :-) So yes having it as breaking change caused by a bug is a good way to state it.

[ruflin]

jenkins, test it

[ruflin]

jenkins, test it

[ruflin]

Thanks for converting this to a table test. It makes it much easier to compare input with output. It makes it a bit harder to write json then before but I think the benefits outweight here.

[exekias]

thank you for taking this!

[exekias]

jenkins, test it please

[ruflin]

@walktall Thanks for your work on this.

@exekias We should make sure this is communicate properly for 6.x in case people had queries on the label keys which will not work anymore.

[vjsamuel]

There should be a better way to handle this as it invokes disparity between how objects are represented in Kubernetes and how autodiscover is configured in Beats. For example a label prometheus.io.scrape would now have to be configured as prometheus_io_scrape. if such conversions need to happen they should ideally happen either in a configurable manner where dedoting can be turned off or later in the pipeline in the form of a processor. Not all backends may need this dedoting to happen and there maybe usecases where everything appears the way it actually is on Kubernetes.

[exekias]

I'm ok with making this optional, will open a PR for it 👍 I want to fix the way we handle metadata with dots too

[walktall]

Maybe it can be done here?https://github.com/elastic/beats/blob/master/libbeat/processors/add_kubernetes_metadata/kubernetes.go#L124

[vjsamuel]

@walktall, autodiscover doesnt use this as it presets all the pod meta. so it would fail for that flow.

[ruflin]

The problem with the dotted keys is that they also caused some issues in the processors. But this should be solved by #5916

I agree we should make it optional as I would also prefer to have the same representation in ES as is delivered by k8s. It creates the potential issue of collisions as also discussed in micrometer-metrics/micrometer#154 (comment) My thinking here is that Metricbeat should have a way prevent such collisions at least for one single event. Meaning if there is {"a": 15, "a.b": 17} that it makes something like {"a.value": 15, "a.b": 17} out of it. Would be great to have a very efficient way of doing that.

+1 on having a processor for it. The important part with a processor is that we can specify which part of an event should be dedotted as in most cases it's only the labels.