some metrics and their origin

[cemo]

I did not want to open an issue for this but could not find really a better way. I am using dashboards of this repo without using prometheus operator and could not find what is filling container_network_receive_bytes_total and container_cpu_usage_seconds_total metrics. These seems as part of kube-proxy but could not find necessary scrapping configurations.

[dannyk81]

I believe they come from cAdvisor which is embedded into kubelet.

[cemo]

which configuration of prometheus is scraping kubelet's?

[brancz]

The kubelets are a tricky one, a few Prometheus Operator features play together for those to get scraped:

• The kubelets are discovered via this ServiceMonitor, from which the Prometheus Operator generates a Prometheus configuration.
• The Prometheus Operator synchronizes the internal IPs of the kubelets into an Endpoints object, from which they are discovered from. The object that it is synchronized into is specified via the --kubelet-object flag.

[brancz]

Closing here for now. Feel free to comment here if you have further questions.

[dkoshkin]

Just found this and running into the same issue, does this metric only work when using the operator, Im trying to get these charts to work using helm charts but not able to :(.

I can see it in prometheus, but its not working in the Grafana chart.

[brancz]

It does not only work with the Prometheus Operator, but it makes is especially easy to scrape them, and with out of the box configuration from kube-prometheus that will work pretty much one-click.

As there is currently just a Prometheus Operator chart I'm unsure I understand which charts you are referring to @dkoshkin .

[dkoshkin]

My bad @brancz I should have specified I was playing around with https://github.com/kubernetes/charts/tree/master/stable/grafana

[cemo]

I explained in other message but might be better to stay at here too. Default scrape interval is 1m and dashboard is using 1m for rate function. So there is not enough data for rate function. Change dashboard interval to 2m or decrease scrape interval to 30s.

[dkoshkin]

Thanks @cemo that worked!