Kube-Prometheus-Top [ kptop ]

A Python tool that provides Monitoring for Kubernetes Nodes, Pods, Containers, and PVCs resources on the terminal through Prometheus metircs

kptop2.mp4

Motivation

The resources metrics provided by the K8s APIs are very limited compared what what's scraped by Prometheus.

This tool is using Prometheus as a data source for metrics to display all the needed informations right on the terminal.

Project Status

• Top Nodes
• Live monitoring for Nodes
• Top Pods
• Live monitoring for Pods/Containers
• Top PVCs

---

Installation

Compatible with Python 3.6+

on PyPi

pip3 install kptop --upgrade

only in this release (forgot to put it in the requirements 😅 )

pip3 install asciichartpy

---

Environment Variables

ENV	Description	Default	Required
KUBE_PTOP_PROMETHEUS_SERVER	Prometheus server URL		Yes
KPTOP_BASIC_AUTH_ENABLED	Whether basic authentication is needed to connect to Prometheus	False	No
KPTOP_PROMETHEUS_USERNAME	Prometheus username		No
KPTOP_PROMETHEUS_PASSWORD	Prometheus password		No
KPTOP_INSECURE	Verify SSL certificate	False	No
KPTOP_NODE_EXPORTER_NODE_LABEL	node exporter "node label"	"node"	NO
KPTOP_START_GRAPHS_WITH_ZERO	By default graphs begin with '0' to let the graph take its full hight	True	NO

CLI Arguments

ENV	Description	Default
--namespace, -n	Specify a Kubernetes Namespace	default
--all-namespaces, -A
--container, -c	Specify a container
--interval, -i	Live monitoring update interval	8 [NOTE: the actuall update depends on the Prometheus scaping interval (15s by default)]
--debug, -d	Enable debugging logging mode	False
--verify-prometheus, -V	Verify connectivity to Prometheus server & check the existence of the needed exporters
--sort-by-mem-usage, -s	Sort top result by memory usage	False

---

Usage

Different ways to connect to Prometheus server:

1. You have direct access to it (Like in dev environments)
2. Prometheus is exposed publically/over-vpn (mostly with an Ingress)
3. You can use kubectl port-forward command
4. You also can run kptop as a Kubernetes pod (to be built in the next release))

export KPTOP_PROMETHEUS_SERVER="http://prometheus.home-lab.com"

Top nodes

kptop nodes

NODE      MEM TOTAL    MEM USAGE    MEM FREE      CPU CORES  CPU USAGE%      RUNNING PODS
worker-1  19.6 gb      16.92 gb     2.69 gb               6  9%                        14
worker-2  19.6 gb      9.52 gb      10.08 gb              6  9%                        27

Live monitoring for Nodes

kptop node <NODE>

Top pods

kptop pods -n <NAMESPACE>

kptop pods -n elk-stack

NAMESPACE    POD                                      MEM LIMIT    MEM USAGE    MEM USAGE %    MEM USAGE MAX    MEM FREE    CPU LIMIT    CPU USAGE
elk-stack    elasticsearch-master-0                   2.0 gb       1.38 gb      68%            2.0 gb           635.25 mb   1000m        0.04m
elk-stack    elasticsearch-master-1                   2.0 gb       1.49 gb      74%            2.0 gb           522.05 mb   1000m        0.03m
elk-stack    strimzi-filebeat-filebeat-f8ms7          200.0 mb     85.73 mb     42%            174.16 mb        114.27 mb   1000m        0.02m
elk-stack    haproxy-ingress-filebeat-filebeat-pq2wf  200.0 mb     87.73 mb     43%            171.31 mb        112.27 mb   1000m        0.03m
elk-stack    strimzi-filebeat-filebeat-r7dht          200.0 mb     119.12 mb    59%            199.52 mb        80.88 mb    1000m        0.02m
elk-stack    haproxy-ingress-filebeat-filebeat-lzqdt  200.0 mb     98.66 mb     49%            199.57 mb        101.34 mb   1000m        0.02m
elk-stack    my-kibana-kibana-79448f7fb7-wf4t6        2.0 gb       342.87 mb    16%            618.07 mb        1.67 gb     1000m        0.02m
elk-stack    my-logstash-logstash-0                   1.5 gb       1008.22 mb   65%            1.21 gb          527.78 mb   1000m        0.02m

kptop pod -n kube-system

NAMESPACE    POD                                              MEM LIMIT    MEM USAGE    MEM USAGE%    MEM USAGE MAX    MEM FREE    CPU LIMIT    CPU USAGE
kube-system  coredns-558bd4d5db-nfcjq                         170.0 mb     26.0 mb      15%           42.77 mb         144.0 mb    ---          0.0m
kube-system  coredns-558bd4d5db-vcstr                         170.0 mb     17.45 mb     10%           24.04 mb         152.55 mb   ---          0.0m
kube-system  etcd-master                                      ---          85.02 mb     ---           391.7 mb         ---         ---          0.02m
kube-system  kube-apiserver-master                            ---          635.97 mb    ---           731.13 mb        ---         ---          0.09m
kube-system  kube-controller-manager-master                   ---          95.55 mb     ---           145.41 mb        ---         ---          0.03m
kube-system  kube-proxy-q6nr7                                 ---          27.46 mb     ---           58.91 mb         ---         ---          0.0m
kube-system  kube-proxy-q489q                                 ---          21.98 mb     ---           63.0 mb          ---         ---          0.0m
kube-system  kube-proxy-bghp6                                 ---          22.35 mb     ---           64.1 mb          ---         ---          0.0m
kube-system  kube-scheduler-master                            ---          37.04 mb     ---           61.68 mb         ---         ---          0.0m
kube-system  nfs-subdir-external-provisioner-b97f4d9f5-bjp2h  ---          9.43 mb      ---           37.48 mb         ---         ---          0.0m

Live monitoring for Pods

kptop pod <POD> -n <NAMESPACE>

Live monitoring for Containers

kptop pod <POD> -n <NAMESPACE> -c <CONTAINER>

Top PVCs

kptop pvcs <NAMESPACE>

NOTE: in this example, all VPCs have the same capacity because this is a testing environment (using nfs-provisioner)

kptop pvcs --all-namespaces

NAMESPACE    PVC                                          VOLUME                CAPACITY    USED      USED %    FREE      FREE %
elk-stack    elasticsearch-master-elasticsearch-master-0  elasticsearch-master  123.14 gb   21.42 gb  17%       95.43 gb  77%
elk-stack    elasticsearch-master-elasticsearch-master-1  elasticsearch-master  123.14 gb   21.42 gb  17%       95.43 gb  77%
elk-stack    elasticsearch-master-elasticsearch-master-2  elasticsearch-master  ?           ?         ?         ?         ?
kafka        data-0-kafka-cluster-region1-kafka-0         data-0                123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-0-kafka-cluster-region1-kafka-1         data-0                123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-0-kafka-cluster-region1-kafka-2         data-0                123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-0-kafka-cluster-region2-kafka-0         data-0                123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-0-kafka-cluster-region2-kafka-1         data-0                123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-0-kafka-cluster-region2-kafka-2         data-0                123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-1-kafka-cluster-region1-kafka-0         data-1                123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-1-kafka-cluster-region1-kafka-1         data-1                123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-1-kafka-cluster-region1-kafka-2         data-1                123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-2-kafka-cluster-region1-kafka-0         data-2                123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-2-kafka-cluster-region1-kafka-1         data-2                123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-2-kafka-cluster-region1-kafka-2         data-2                123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-3-kafka-cluster-region1-kafka-0         data-3                123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-3-kafka-cluster-region1-kafka-1         data-3                123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-3-kafka-cluster-region1-kafka-2         data-3                123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-kafka-cluster-region1-zookeeper-0       data                  123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-kafka-cluster-region2-zookeeper-0       data                  123.14 gb   21.42 gb  17%       95.43 gb  77%
prometheus   my-prometheus-alertmanager                   storage-volume        123.14 gb   21.42 gb  17%       95.43 gb  77%
prometheus   my-prometheus-server                         storage-volume        123.14 gb   21.42 gb  17%       95.43 gb  77%

Verify Prometheus connectivity

kptop --verify-prometheus

Sample output

Verifying Prometheus connection: Connected                     
{
  "connected": true,
  "status_code": 200,
  "reason": "",
  "fail_reason": ""
}

Verifying Prometheus Exporters:

* Node Exporter:  Found             
{
  "success": true,
  "fail_reason": "",
  "result": {
    "found_versions": {
      "1.3.1": "2"
    }
  }
}

* Kubernetes Exporter:  Found           
{
  "success": true,
  "fail_reason": "",
  "result": {
    "found_git_versions": {
      "v1.21.0": "3",
      "v1.21.14": "1"
    }
  }
}

---

Logging

Default log file location is "/tmp/kptop.log"

---

Known Issues

[1] Node Exporter metrics don't return data

• This is NOT an issue, the node exporter NODE label change from version to another, currently we encountered only "kubernetes_node" or "node"
• "node" is the default, to fix the it you can change it with the "KPTOP_NODE_EXPORTER_NODE_LABEL" Environment variables

export KPTOP_NODE_EXPORTER_NODE_LABEL="node" # default
export KPTOP_NODE_EXPORTER_NODE_LABEL="kubernetes_node"

auto detection of exporters verstions can be implemented later (if needed).

---

Reach me anytime on Linkedin