Kubernetes manifest shouldn't use hostNetwork with autodiscovery

[TomaszKlosinski]

Hi, using the official k8s manifest with autodiscovery enabled is causing:

2019-12-09T15:57:46.777Z	ERROR	kubernetes/util.go:97	kubernetes: Querying for pod failed with error: pods "k8s-cluster-worker-02" not found

due to this line:
https://github.com/elastic/beats/blob/master/deploy/kubernetes/filebeat-kubernetes.yaml#L63

Because then it sets the hostname of the pod equal to the hostname of the host machine. And this hostname is picked up by kubernetes client in libbeat:
https://github.com/elastic/beats/blob/master/libbeat/common/kubernetes/util.go#L89

Removing these lines helps:

      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet

But the proper solution would be reconfiguring the k8s client in libbeat to fetch pod name differently. Pod's hostname is not stable across deployments.

[exekias]

Hi @TomaszKlosinski, this line should avoid that problem: https://github.com/elastic/beats/blob/7.5/deploy/kubernetes/filebeat-kubernetes.yaml#L26, are you passing the host setting to autodiscover?

[TomaszKlosinski]

I've tried but it didn't make any effect, the pod's hostname was still the same a host's hostname. In fact, if I kubectl exec to the pod and run "echo ${NODE_NAME}" it returns the host's hostname.

[willemm]

I had the same problem, but it was caused because the example config has 'node: ${NODE_NAME}' which I had to change to 'host: ${NODE_NAME}'

[TomaszKlosinski]

Interesting. I'm pretty sure, I've tried both node and host and it didn't work, but I can test it again.

More general question: what's the point of using host network in this case?

[dalibor-aleksic-atomia]

Same question:

More general question: what's the point of using host network in this case?

Can these lines be safely removed?

hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet

Using hostNetwork also conflicts with Calico Network Policies: https://github.com/projectcalico/felix/issues/1361

[TomaszKlosinski]

Can these lines be safely removed?

I can share my opinion as a user of filebeat. I did remove them and my filebeat runs as expected since late November, 2019. I haven't encountered any problems.

[exekias]

HI everyone,

We use hostNetwork in filebeat to make sure we can access hostname and some other host related metadata. We use this for enhanced experience in the UI.

That said, it's not mandatory for Filebeat to work, and you can remove hostNetwork setting without issues. The only important bit is that you pass the right host setting to add_kubernetes_metadata/ kubernetes autodiscover provider. This will guarantee they are able to filter metadata to the right node.

I'm closing this issue as we don't have plans to remove hostNetwork setting, for the reasons stated above.