Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Metrics] Extend metrics to measure physical performance stats #421

Open
dcrankshaw opened this Issue Feb 28, 2018 · 3 comments

Comments

Projects
None yet
3 participants
@dcrankshaw
Copy link
Contributor

dcrankshaw commented Feb 28, 2018

It would be great to extend our monitoring infrastructure to measure more physical performance. To start with, @blackhat06 suggested tracking the following resource metrics:

  • Disk IO: % time that device was busy
  • Memory: % of total memory capacity in use
  • CPU Utilization: 5,10,15 min
  • Memory Utilization: Breakdown by memory
  • % volume usage: Disk all mounted
  • Bits IN/Out (ethernet)
  • Volume I/O
  • Process count / running/blocked
@simon-mo

This comment has been minimized.

Copy link
Contributor

simon-mo commented Feb 28, 2018

Prometheus can track these with node exporter https://github.com/prometheus/node_exporter/blob/master/README.md

For Kubernetes we can just scrape kube-api-server/metrics. Kubernetes expose Prometheus metrics there

@simon-mo simon-mo changed the title Extend metrics to measure physical performance stats [Metrics] Extend metrics to measure physical performance stats Mar 10, 2018

@simon-mo

This comment has been minimized.

Copy link
Contributor

simon-mo commented Mar 10, 2018

Update:

  • For Docker, we can safely assume user only has on node so we can just run a node exporter at startup.
  • Kubernetes's API service does expose metrics but the metrics are about the api server requests itself and etcd usage. We should use a DaemonSet. It will deploy a prometheus node exporter to each node (https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/). In fact this is recommended practice by Kubernetes.
@dcrankshaw

This comment has been minimized.

Copy link
Contributor Author

dcrankshaw commented Mar 12, 2018

Awesome. cc @blackhat06

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.