Report host metadata for Kubernetes logs

[exekias]

Filebeat was not reporting host metadata in the default Kubernetes manifest,
this change gives Filebeat access to the hostNetwork to retrieve
localhost metadata. add_host_metadata is added to gather it.

After this change host metadata should be present in logs, which will make it easier to correlate them with host metrics.

[exekias]

Wondering if we should go this way or just report host.name from kubernetes metadata: #10926, the added benefit of doing it there is that it should always work, regardless the deployment model, on the other side, it will report less info (hostname only, no other network/os/arch data)

[odacremolbap]

kubernetes node info is partly managed by the cloud controller manager.
there is no guarantee that you get the same name when you use retrieve the hostname from the OS API than when you use the kubernetes API.

iirc for AWS kubernetes would return the FQDN, it is up to cloud providers to fill the node information (including IPs) with whatever information they find relevant.

That said, I think the kubernetes API contains all data needed to correlate with other non kubernetes events if needed. Also, I haven't looked at how beats retrieve node info but Node Status contains some of the info you mention above.

Without having a strong opinion, as a user I would like to have the kubernetes API info to avoid multi-sources, but I wouldn't put the info into host.name since beats seem to allow add_host_metadata alongside any other metric that might be trying to re-inform host related fields.

Is this kind of overlapping something we have faced before?

[exekias]

kubernetes node info is partly managed by the cloud controller manager.
there is no guarantee that you get the same name when you use retrieve the hostname from the OS API than when you use the kubernetes API.

iirc for AWS kubernetes would return the FQDN, it is up to cloud providers to fill the node information (including IPs) with whatever information they find relevant.

That's a fair point, if we use different codepaths between Metricbeat and Filebeat we could end up with different values -> problems.

That said, I think the kubernetes API contains all data needed to correlate with other non kubernetes events if needed. Also, I haven't looked at how beats retrieve node info but Node Status contains some of the info you mention above.

Without having a strong opinion, as a user I would like to have the kubernetes API info to avoid multi-sources, but I wouldn't put the info into host.name since beats seem to allow add_host_metadata alongside any other metric that might be trying to re-inform host related fields.

I see some benefits from extracting all host metadata from Kubernetes, that link looks promising. For instance, Filebeat should not need to have access to the host network just to retrieve the hostname.

Is this kind of overlapping something we have faced before?

We had situations like this, for instance, add_cloud_metadata won't overwrite existing info:

beats/libbeat/processors/add_cloud_metadata/add_cloud_metadata.go

Line 369 in d21cf68

if !p.initData.overwrite {

, where some modules report cloud metadata for the service they are monitoring (ie AWS)

I'm currently leaning towards the approach on this PR, for some reasons:

• If we have a single agent in the future, it will require the permissions anyway
• This change makes sure we use the same code paths both for Metricbeat and Filebeat, so data must be the same
• add_host_metadata should add uniform data across k8s & non k8s scenarios, and UI uses it heavily

This should not stop any other research we can do in the future to revisit how metadata is added.

WDYT?

[odacremolbap]

fully agree

[odacremolbap]

SGTM if that's so to CI