[Metricbeat] Access Kubernetes metrics from somewhere other than the read-only kublet port

[dpankros]

Currently, it appears that metricbeat fetches metrics for kubernetes from the read-only http-metrics port (10255). Using this port is now deprecated. Kubernetes metrics need to be fetched through some other mechanism in the near future.

To make matters worse, the AWS EKS AMI has dropped support for the readonly port in recent versions. Out of the box, metricbeat no longer works on EKS without doing some custom work to the kubelet on the node. We upgraded recently to gain the benefit of other fixes implemented in the recent AMIs (like logrotate) and now we don't see any pod metrics.

This is somewhere between a bug and an enhancement.

[odacremolbap]

Hi @dpankros ,

metricbeats do not enforce the read only kubelet port, you can use the commented settings here

https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-kubernetes.html#_example_configuration_17

if executing as a daemonset something like this should work on a regular kubernete cluster

    - module: kubernetes
      metricsets:
        - node
        - system
        - pod
        - container
        - volume
      period: 10s
      host: ${NODE_NAME}
      hosts: ["https://${NODE_NAME}:10250"]
      enabled: true
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      ssl.verification_mode: none
      ssl.certificate_authorities:
        - /var/run/secrets/kubernetes.io/serviceaccount/ca.crt

[odacremolbap]

@dpankros we will be closing this because of inactivity, if the comment above didn't solve the issue, please re-open

[salt-mountain]

@odacremolbap Hopefully this comment is not too much of a resurrection, but I have a question regarding the page and what you posted. Those seem to be the instructions for running metricbeat as a pod within the cluster.

But what about the correct configuration for running it outside of the cluster? Simply using in_cluster: false and specifying the kube_config doesn't seem to cut it. There obviously is now nothing listening to on the port specified in the documentation.

[PhaedrusTheGreek]

In master we have changed this to use the secured port by default. I'm wondering if it would be worthwhile to backport this at least to 7.6. Those three lines tested working for me under 7.5.

[jlind23]

Backlog grooming: Closing it for now until further activity, can still be reopened if needed.